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Executive Summary 


The state board is required to “institute a system for the coordination and delivery of 
high-quality professional development,” (W. Va. Code §i8-2l-i), including defining goals and 
standards for professional development, and roles and responsibilities for state and regional 
professional development providers. Accordingly, the board is required to establish an annual 
master plan for professional development in public schools across the state, to include the 
“state board-approved plans for professional development by the State Department of Educa- 
tion, the Center for Professional Development, the state institutions of higher education and 
the regional educational service agencies to meet the professional development goals of the 
state board” (W. Va. Code §18-21-4). The state board is also required to establish processes for 
evaluating the “effectiveness, efficiency, and impact of the statewide professional development 
plan” and submit its report to the Legislative Oversight Commission on Education Accounta- 
bility” (W. Va. Code §18-21-4). As in previous years, the board charged the West Virginia De- 
partment of Education’s (WVDE) Office of Assessment and Research with the responsibility 
of conducting the evaluation. 

In response to these requirements, the goals of this evaluation are to study the effec- 
tiveness, efficiency, and impact of the statewide master plan for professional development (PD 
Master Plan) by investigating seven questions: 

With regard to effectiveness, 

EQi. What was the level of implementation for the PD Master Plan, overall and by indi- 
vidual providers? 

EQ2. What were the impediments, if any, to its full implementation? 

EQ3. To what extent did providers’ offerings reflect the WVBE Standards for Profes- 
sional Learning? 

EQ4. To what extent did providers’ offerings reflect research-based professional devel- 
opment practices? 

With regard to efficiency, 

EQ5. To what extent did providers collaborate in the delivery of professional develop- 
ment, thereby reducing duplicative efforts? 

With regard to impact, 

EQ6. How well did providers’ offerings address the WVBE’s 2013-2014 Goals for Profes- 
sional Learning? 

EQ7. What was the impact of the professional development offered through the 2013- 
2014 PD Master Plan on educators’ knowledge, practice, and attitudes? 


VII 


Executive Summary 


Methods 


Population studied 

The unit of analysis for this study was the individual provider required by W. Va. Code 
§18-21 to participate in the PD Master Plan. This provider group included the following or- 
ganizations and agencies: 

• Center for Professional Development (CPD) 

• Two of the required ten public institutions of higher education (IHEs) with teacher 
education programs 

o Fairmont State University 

o Marshall University (two programs: Clinical Studies and Professional Develop- 
ment Schools and the June Harless Center) 

• All eight regional education service agencies (RESAs) 

• Eleven offices in the West Virginia Department of Education (WVDE) that provide 
professional learning experiences to educators across the state, including the Offices 
of 

o Assessment, Accountability, and Research 
o Career and Technical Accountability and Support 
o Career and Technical Instruction 
o Early Learning 
o Federal Programs 
o Institutional Education Programs 
o Instructional Technology 
o Professional Preparation (Certification) 
o School Improvement 
o Secondary Learning 
o Special Programs 

To investigate the performance of these 23 organizations and agencies we surveyed the 
participants in the professional development sessions they offered, to gain their views about 
the quality of their learning experiences. This population included, among others, district cen- 
tral office staff, school administrators, general and special education teachers, instructional 
support teachers, school librarians/media specialists, and paraprofessionals. 

Research design 

To investigate the seven evaluation questions, we collected data from providers in the 
form of session reports, which required providers to report for each session they conducted, 
such information as the alignment of the session with the seven state Board Standards for 
Professional Learning, the beginning and ending dates, the duration and attendance for the 
session, its format and county location, and e-mail addresses for participants. The reporting 
year was divided into three collection periods: July 1 through November 30, 2013, December 
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l, 2013 through April 30, 2014, and May 1 through June 30, 2014. E-mail addresses submitted 
during the first two data collection periods were used in a survey of participants conducted 
in two parts, with one random sample of unique participants surveyed in January 2014 and a 
second sample in June 2014. It should be noted that participants in professional development 
that took place during the third data collection period (May l-June 30, 2014) were not sur- 
veyed because of the difficulties involved in collecting data from educators during the summer 
months. 

In addition to the session reports and the participant survey data, extant documents 
were used, especially Board policies, in order to put certain aspects of the study into context. 

Findings 

Major findings for 2013-2014 include the following, arranged here by effectiveness, 
efficiency, and impact: 

Effectiveness of the Master Plan 

• The PD Master Plan included more topics (479), sessions (1,056), and participants 
(33,196) than in previous years. 

• Regarding the effectiveness of providers in meeting the Board Standards for Profes- 
sional Learning, overall, the standards that providers most often believed they had met 
were Standard 7 (“Aligns its outcomes with educator performance and student curric- 
ulum standards”) and Standard 5 (“Integrates theories, research, and models of hu- 
man learning into learning designs to achieve its intended outcomes”). The two 
standards providers least often reported meeting were Standard 4 (“Uses a variety of 
sources and types of student, educator, and system data to plan, assess, and evaluate 
professional learning”) and Standard 6 (“Applies research on change and sustains sup- 
port for implementation of professional learning for long-term change”). 

• Regarding the extent to which providers’ offerings reflected five research -based pro- 
fessional development practices, overall, there was the greatest agreement that the 
professional development had focused on content and content pedagogy. Active 
learning scored lowest; second lowest was respondents' estimation that the session 
had provided sufficient duration and timespan to allow them to apply what they were 
learning. Participants were largely in agreement that the professional development 
had been well aligned ( coherent ) with their own needs and those of the school and 
district, and that it had been delivered to allow for participation with colleagues ( col- 
lective participation ). 

• Providers’ session reports seemed to confirm the findings about duration and 
timespan above. Of the 33,000 attendees reported by providers, about 5,700 educators 
participated in professional development lasting at least 30 hours; another 5,400 had 
from 14 to 29 contact hours— durations shown by research to be the minimum needed 
to change teacher practice and impact student learning. The remaining two-thirds of 
participants attended sessions ranging from 1 to 13 hours. About half of all sessions 
began and ended on the same day. 

Efficiency of the Master Plan 
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• The Legislature's call for decentralization of professional development seemed to be 
reflected in the trends for the four major providers in the Plan, with the WVDE de- 
creasing their number of offerings and participants from the previous year, while the 
RESAs' slate of offerings rose dramatically, as did their participant counts. CPD con- 
tinued its trend upward for both topics and participants, while IHES remained stable. 
Still, the WVDE reported the greatest number of both sessions held and participants 
in attendance. 

• RESAs and WVDE offices operated very collaboratively, partnering with each other 
and IHEs. According to CPD session reports, they worked almost completely inde- 
pendently of other providers. It should be noted, however, that CPD, like the RESAs, 
seeks input from the WVDE and others when setting its slate of offerings. 

Impact of the Master Plan 

• While all four of the Board Goals for Professional Learning received coverage, Goal 2 
(“Increase deep content knowledge and proficiency in designing and delivering stand- 
ards-driven instruction and assessments”) was the focus of about 80% of sessions (n 
= 858) with 73% of attendees (n = 24,233). 

• The participant surveys showed a high level of general agreement— at least 75%— that 
the sessions they attended had been helpful in meeting the Board goal with which it 
was aligned. 

• According to retrospective self-reports, the professional development had large effects 
on educators’ knowledge of the PD topic, and moderate effects on their practice and 
attitudes/beliefs. 

Limitations of study 

The response rate for this study was lower than usual (50.5%), which may have intro- 
duced sampling bias. Further, respondents were asked to recall PD sessions they had partici- 
pated in at some point in the past— up to 5 months prior to the survey— so there is a possibility 
of temporal bias. The use of a retrospective pretest/posttest methodology to assess changes in 
knowledge, behavior and skills, and attitudes and beliefs poses some concerns, including the 
possibility of inflated estimations of impacts on respondents’ knowledge, practice, and atti- 
tudes/beliefs. 

Conclusion and Recommendations 

Based on these findings, we make the following recommendations. 

In keeping with the Board Standards for Professional Learning, 

• Increase use of a variety of sources and types of student, educator, and system data to 
plan, assess, and evaluate professional learning, and 

• Increase application of research on change and sustain support for implementation of 
professional learning for long-term change. 


With regard to the use of research-based PD practices, 
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• Increase active learning during professional development sessions, and 

• Provide sufficient duration (30 or more hours) and timespan (weeks or months) to 
allow participants opportunities to apply what they are learning. 

Based on factors present in the larger context of professional development in the state, we 
recommend that the West Virginia State Board of Education and Department of Education 

• Bring all ten of the public IHEs with teacher education programs into the PD Master 
Plan (two participated in 2013-2014). 

• Promote the Board’s Standards for Professional Learning at the school and district 
level, so they will better guide educators’ planning. 

• Consider adoption of a model for professional development providers that aligns with 
and supports local learning communities working to adopt the Board’s Standards for 
Professional Learning. 

• Revisit the purposes and possible uses of the PD Master Plan to guide professional 
learning— aligning it with the Transforming Professional Development Initiative. 

• Consider ways to put the evaluation of the PD Master Plan to better use— reshaping 
the evaluation and its purposes as the plan itself is reshaped. 
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Introduction 


In 2013, the West Virginia Legislature passed and the Governor Earl Ray Tomblin 
signed new legislation requiring broad reforms resulting, in part, from the Governor’s Educa- 
tion Efficiency Audit of West Virginia’s Primary and Secondary Education System (Public 
Works, 2012). This report urged decentralization of the state’s public education system, with 
more authority and responsibility transferring back to regional education service agencies 
(RESAs), districts, and schools. At the same time, the Legislature recognized the need for 
“clear state-level leadership for professional development for all West Virginia public school 
educators and administrators” (W. Va. Code §18-21-1). Accordingly, it continued the require- 
ment that the state board “institute a system for the coordination and delivery of high-quality 
professional development,” (W. Va. Code §18-21-1), including defining goals and standards 
for professional development, and roles and responsibilities for state and regional profes- 
sional development providers. The state board was required to establish an annual master 
plan for professional development in public schools across the state, to include the “state 
board-approved plans for professional development by the State Department of Education, 
the Center for Professional Development, the state institutions of higher education [with 
teacher education programs] and the regional educational service agencies to meet the pro- 
fessional development goals of the state board” (W. Va. Code §18-21-4). 

The state board was also required to establish processes for evaluating the “effective- 
ness, efficiency, and impact of the statewide professional development plan” and to submit its 
report to the Legislative Oversight Commission on Education Accountability” (W. Va. Code 
§18-21-4). The state board, as in the previous three years, charged the West Virginia Depart- 
ment of Education’s (WVDE) Office of Assessment and Research with the responsibility of 
conducting the evaluation. 


Goals of the Evaluation 

The goals of this evaluation are to study the effectiveness, efficiency, and impact of the 
statewide master plan for professional development (PD Master Plan) by investigating seven 
questions: 

With regard to effectiveness, 

EQl. What was the level of implementation for the PD Master Plan, overall and by indi- 
vidual providers? 

EQ2. What were the impediments, if any, to its full implementation? 

EQ3. To what extent did providers’ offerings reflect the WVBE Standards for Profes- 
sional Learning? 

EQ4. To what extent did providers’ offerings reflect research-based professional devel- 
opment practices? 

With regard to efficiency, 

EQ5. To what extent did providers collaborate in the delivery of professional develop- 
ment, thereby reducing duplicative efforts? 


1 


Introduction 


With regard to impact, 

EQ6. How well did providers’ offerings address the WVBE’s 2013-2014 Goals for Profes- 
sional Learning? 

EQ7. What was the impact of the professional development offered through the 2013- 
2014 PD Master Plan on educators’ knowledge, practice, and attitudes? 

Relevant Scholarship 

In 2012, the West Virginia Board of Education adopted the Learning Forward Stand- 
ards for Professional Learning, paraphrased in the PD Master Plan as follows: 

Professional learning that increases educator effectiveness and results for all stu- 
dents— 

• Occurs within learning communities committed to continuous improvement, 
collective responsibility, and goal alignment. 

• Requires skillful leadership to develop capacity, advocate, and create support 
systems for professional learning. 

• Requires prioritizing, monitoring, and coordinating resources for educator 
learning. 

• Uses a variety of sources and types of student, educator, and system data to plan, 
assess, and evaluate professional learning. 

• Integrates theories, research, and models of human learning into learning de- 
signs to achieve its intended outcomes. 

• Applies research on change and sustains support for implementation of profes- 
sional learning for long-term change. 

• Aligns its outcomes with educator performance and student curriculum stand- 
ards. (Learning Forward, n.d.; West Virginia Board of Education, 2013 ). 

The standards were carefully researched in a comprehensive study of professional 
learning, conducted by a team of researchers from Stanford University led by Linda Darling - 
Hammond, with contributions and support from 40 professional associations. Working to- 
gether, they make up components of a system of professional learning, which to be effective 
“most often occurs in learning communities; is supported by strong leadership and appropri- 
ate resources; is drawn from and measured by data on students, educators, and systems; ap- 
plies appropriate designs for learning; has substantive implementation support; and focuses 
on student and educator outcomes” (Mizell, Hord, Killion, & Hirsh, 2011, p. 13). The Stand- 
ards for Professional Learning, in other words, outline the context (learning communities, 
leadership, and resources), educator learning processes (data, learning designs, and imple- 
mentation), and content (outcomes) needed for professional learning to result in improved 
practice and student outcomes. 

Because these standards are meant to guide professional learning in schools and dis- 
tricts at the local level, they present a challenge to statewide and regional providers who must 
align their offerings with them. The standards are meant to guide teacher- and administrator- 
driven professional learning locally— which fits well with the efforts of the legislature and state 
board to provide greater autonomy and authority to schools and districts— while providers in 
the PD Master Plan must put together “plans/offerings” to support the state board’s goals in 
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a statewide system of professional development (W. Va. 
Code §18-21-4; see box for a listing of the 2013-2014 Goals 
for Professional Learning). The Board Goals for Profes- 
sional Learning reflected the priorities for the state educa- 
tion system, which was in its final year before the full 
implementation of the Next Generation state standards and 
the new teacher evaluation system. In 2013-2014 as in pre- 
vious years, the master plan took the form of a slate of ses- 
sion topics, which were planned and implemented by the 
four major provider groups. 

The state standards also posed challenges for the 
evaluation of the PD Master Plan. In prior years, this eval- 
uation has investigated the fidelity of implementation of the 
plan, its effectiveness in supporting the designated Board 
goals, the quality of the experience for participants, and a 
retrospective pre-post participant self-assessment of 
changes in their knowledge, practice, and attitudes /beliefs. 
Due to the urgent and serious focus of the state board and 
legislature on improving professional learning experiences 
for educators across the state, the need for a review of the 
research literature became apparent. 

The literature review (Hammer, 2013) revealed 
both contextual and quality issues to be considered in facil- 
itating professional learning experiences for teachers and 
administrators. These findings fit well with the Learning 
Forward standards. Context matters, in that, professional 
development that is not supported by multiple components 
making up a system of support will likely have minimal im- 
pact on educators and students. Based on their studies in 
mathematics education and the broader education research 
literature, Cobb and Jackson (2011, p. 12) outlined a system 
that includes multiple elements, all working together: 

• Explicit goals for students’ learning 

• A detailed vision of high-quality instruction that 
specifies particular instructional practices that will 
lead to students’ attainment of the learning goals 

• Instructional materials and associated tools de- 
signed to support teachers’ development of these 
practices 


2013-2014 Board 
Goals for 
Learning 

Professional development 

for the 2013-2014 school 

year shall: 

1. Increase the 
knowledge and skills 
of all pre-K educators 
to deliver a compre- 
hensive preK-third 
grade approach to 
early childhood edu- 
cation that includes a 
balanced approach to 
early literacy. 

2. Increase deep content 
knowledge and profi- 
ciency in designing 
and delivering stand- 
ards-driven instruc- 
tion and assessments 
for all preK-12 West 
Virginia educators. 

3. Improve leadership 
competencies for 
principals and assis- 
tant principals in or- 
der to support high 
quality teaching and 
learning. 

4. Support the full imple- 
mentation of the re- 
vised educator 
evaluation system. 


• District teacher professional development that focuses on the specific practices, is or- 
ganized around the above materials, and is sustained over time 
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• School-based professional learning communities (PLCs) that provide ongoing oppor- 
tunities for teachers to discuss, rehearse, and adapt the practices that have been intro- 
duced in district professional development 

• Classroom assessments aligned with the goals for students’ learning that can inform 
the ongoing improvement of instruction and the identification of students who are 
currently struggling 

• Additional supports for struggling students to enable them to succeed in mainstream 
classes. 

The literature review also revealed a widespread consensus about specific qualities of effective 
professional development— qualities that fit well with the fourth and fifth bullet items above 
and also aligned with the Learning Forward standards focused on educator learning pro- 
cesses (data, learning designs, and implementation), and content (outcomes). Within this con- 
text, research has shown that effective professional development tends to have the following 
elements: 

• Content and content pedagogy focus. This element includes both deepening teachers’ 
knowledge of the subject matter they are teaching and the pedagogical approaches that 
have been shown to be successful in helping students learn that subject matter. Effec- 
tiveness is improved if the professional development uses the curriculum materials 
that teachers will later use with their students (Blank, de las Alas, & Smith, 2008; Car- 
penter et al., 1989; Clewell et al., 2004; Cohen & Hill, 1998, 2001; Desimone, Porter, 
Garet, Yoon, & Birman, 2002; Desimone, Smith, & Phillips, 2013; Doppelt et al., 2009; 
Garet et al., 2001; Kennedy, 1998; McCutchen et al., 2002; Penuel, Fishman, Yaga- 
muchi, & Gallagher, 2007; Yoon et al., 2007). 

• Coherence. This element involves providing professional development experiences in 
a progression that builds on previous experiences and aligns with school goals and 
with state standards, curriculum, and assessments. Coherent professional develop- 
ment programs encourage continuing professional communication among teachers, 
either in their own school or with others in the district who teach similar subject matter 
or students (Cohen & Hill, 1998; Desimone et al., 2002; Garet et al., 2001; Grant, Pe- 
terson, & Shojgreen-Downer, 1996; Lieberman & McLaughlin, 1992). 

• Active learning. Opportunities for active learning can include reviewing student data 
and work, practicing a new skill and obtaining feedback, planning how new curriculum 
materials and new teaching methods will be used in the classroom, and engaging in 
discussions and in written work (Desimone et al., 2002; Garet et al., 2001; Penuel, 
Fishman, Yagamuchi, & Gallagher, 2007). 

• Collective participation. Professional development that has collective participation of 
teachers from the same school, department, or grade helps increase opportunities to 
discuss concepts, skills, and problems that arise when teachers work to integrate what 
they have learned into their classroom practice. Over time, it can lead to a professional 
culture— or learning communities— in which teachers in a school develop a common 
understanding of instructional goals, methods, problems, and solutions— an under- 
standing that is sustained over time, even when some teachers leave and others join 
the group (Desimone et al., 2002; Desimone, Smith, & Ueno, 2006; Garet, et al., 2001; 
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Johnson, Kahle, & Fargo, 2007; Penuel, Fishman, Yagamuchi, & Gallagher, 2007; 
Saunders, Goldenberg, & Gallimore, 2009). 

• Duration , including time span and contact hours. Depending on the complexity and 
difficulty of the knowledge and skills teachers are learning, the number of contact 
hours may vary, but research suggests that at least 30 hours are needed to impact stu- 
dent achievement. Sustaining the experience over one or more school years is also im- 
portant, allowing for more opportunity for teachers to try out new practices and benefit 
from additional feedback and communication with trainers, coaches, or colleagues in 
professional learning communities in their schools (Blank, de las Alas, & Smith, 2008; 
Clewell et al., 2004; Yoon et al., 2007). 

This evaluation focuses on these qualities in its measures of effectiveness due to the 
identified qualities’ support of the Learning Forward standards, and their relevance to the 
realm of influence within which statewide and regional providers work. 

Methods 

To investigate the seven evaluation questions, we collected data from providers in the 
form of session reports, which required providers to report for each session they conducted, 
the alignment of the session with the seven state Board Standards for Professional Learning, 
the beginning and ending dates, the duration and attendance for the session, its format and 
county location, and e-mail addresses for participants. The reporting year was divided into 
three collection periods: July 1 through November 30, 2013, December 1, 2013 through April 
30, 2014, and May 1 through June 30, 2014. E-mail addresses submitted during the first two 
data collection periods were used in a survey of participants conducted in two parts, with one 
random sample of unique participants surveyed in January 2014 and a second group in June 
2014. It should be noted that participants in professional development that took place during 
the third data collection period (May l-June 30, 2014) were not surveyed because of the dif- 
ficulties involved in collecting data from educators during the summer months. 

In addition to the session reports and the participant survey data, extant documents 
were used, especially Board policies, in order to put certain aspects of the study into context. 

Population Characteristics 

The unit of analysis for this study was the individual provider required by W. Va. Code 
§l8-2l to participate in the PD Master Plan. This provider group included the following or- 
ganizations and agencies: 

• Center for Professional Development 

• Two of the required ten public institutions of higher education (IHEs) with teacher 
education programs 

o Fairmont State University 

o Marshall University (two programs participated separately, Clinical Experiences 
and Professional Development Schools and the June Harless Center) 

• All eight regional education service agencies (RESAs) 
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• Eleven offices in the West Virginia Department of Education (WVDE) that provide 
professional learning experiences to educators across the state, including the Offices 
of 

o Assessment, Accountability, and Research 
o Career and Technical Accountability and Support 
o Career and Technical Instruction 
o Early Learning 
o Federal Programs 
o Institutional Education Programs 
o Instructional Technology 
o Professional Preparation (Certification) 
o School Improvement 
o Secondary Learning 
o Special Programs 

To investigate the performance of these 23 organizations and agencies we surveyed the 
participants in the professional development sessions they offered to gain their views about 
the quality of their learning experiences. This population included, among others, district cen- 
tral office staff, school administrators, general and special education teachers, instructional 
support teachers, school librarians/media specialists, and paraprofessionals. 

Sampling Procedures 

For both the first and second participant surveys, we applied multistage sampling- 
systematic, stratified, and simple random— to select participants for this study, using the fol- 
lowing procedure: 

• We combined the session participant e-mail addresses submitted in providers’ session 
reports— each e-mail address with its associated PD Master Plan session ID and pro- 
vider— into one comprehensive Excel file (N = 12,621 for the first participant survey; 
N = 9,024 for the second). 

• In the second survey, email addresses that were included in the first survey sample 
were removed. 

• Participants were sorted by e-mail address and assigned a random number. The sam- 
ple was then resorted by random number and the first occurrence of each individual’s 
e-mail address was selected, resulting in 8,285 unique email addresses for the first 
data collection period, and 4,608 for the second. 

• The data were then stratified by provider. Knowing the population of each provider, 
the MaCorr Research Sample Size Calculator 1 was used to determine the sample size 
needed to be between a +/~ 3 % confidence interval at the 95% confidence level. To that 
figure, an additional 50% was added when possible, to allow for attrition and lack of 


Available online at http://www.macorr.com/sample-size-calculator.htm. 
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response. A simple random sample was then drawn for each provider. For some pro- 
viders reporting lower numbers of session participants, the entire population of 
unique e-mail addresses was included in the sampling. A total of 9,416 unique e-mail 
addresses were included in the sample. Of those 582 bounced back when the e-mail 
communications began, leaving a sample of 8,834 viable e-mail addresses. 

Measures 

To address the seven evaluation questions, different combinations of the two primary 
data sources described above and policy documents related to the Center for Professional De- 
velopment and RESAs were employed, as outlined in Table 1 (next page). 
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Table 1. 2013-2014 PD Master Plan Evaluation Questions, Indicators, and Data Sources: Alignment with 

Legislative Mandate 


Legislative 

Mandate 

Evaluation Question Indicators Data Sources 

Effectiveness 

EQ1. What was the level of • Percentage of PD offerings that were 2013-2014 PD Master 

implementation for the PD planned versus the ones delivered Plan Session Report 

Master Plan, overall and by • Participation in the evaluation of the PD database (online 

individual providers? Master Plan system) 

• Number of participants served 

EQ2. What were the Reasons given by providers for why some Providers' responses 

impediments, if any, to its full planned sessions were not held to query 

implementation? 

EQ3. To what extent did Proportion of PD offerings that address 2013-2014 PD Master 

providers' offerings reflect the each of the seven standards: (a) learning Plan Session Report 

WVBE Standards for communities, (b) leadership, (c) resources, database (online 

Professional Learning? (d) use of data, (e) learning designs, (f) system) 

change and implementation, and (g) 
outcomes 

EQ4. To what extent did Proportion of PD offerings that had the 2013-2014 PD Master 

providers' offerings reflect following elements: Plan Session Report 

research-based professional • Content and content pedagogy focus; database (online 

development practices? • Coherence with teachers' professional system) (duration/ 

needs, school goals, and state standards, timespan) 
curriculum, and assessments; WVBE pD Master p|an 

. Active learning, including time for Participant Survey 

planning implementation; 2013-2014 

• Collective participation of teachers or 
administrators from the same district, 
school, grade level, content area, or 
specialization; and 

• Duration (at least 30 hours) and timespan 
(over months or years) 

Efficiency 

EQ 5. To what extent did Number of sessions that were offered in 2013-2014 PD Master 

providers collaborate in the partnerships among the PD Master Plan Plan Session Report 

delivery of professional providers database 

development? 

Board policy 
documents for RESAs 

and CPD 

Impact 

EQ6. How well did providers' • Proportion of PD offerings targeting each 2013-2014 PD Master 
offerings address the WVBE's of the goals overall Plan Provider Session 

2013-2014 Goals for • Participant reports of PD's helpfulness in Submissions database 

Professional Learning? meeting board goals (online form) 

WVBE PD Master Plan 
Participant Survey 
2013-2014 

EQ7. What was the impact of Participant views of the impact of the PD on WVBE PD Master Plan 
the professional development their own knowledge, practice, and Participant Survey 

offered through the 2013- attitudes 2013-2014 

2014 PD Master Plan on 
educators' knowledge, 
practice, and attitudes? 
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The online questionnaire used in the participant survey was similar to the one used in 
previous years with one significant exception. The quality measure used in previous years was 
replaced by a more comprehensive measure based on the findings of the literature review de- 
scribed in the Relevant Scholarship section above (page 2). Three questions were developed 
for each of five quality indicators— that is, content focus, coherence, active learning, collective 
participation, and duration/timespan— to form a Research-Based PD Practices Index (See Ap- 
pendix A). 

The online session report, which providers filled out to report each of the sessions they 
delivered, was similar to previous years. Policy and state code outlining the responsibilities of 
RESAs and the Center for Professional Development, used to understand contextual issues, 
especially with regard to collaboration, were accessed via the websites for the state board 
(http://wvde.state.wv.us/policies/) and the West Virginia State Legislature 
(http://www.legis.state.wv.us/wvcode/Code.cfm?chap=l8&art=i). 

Research Design 

Descriptive statistics were employed for five of the seven evaluation questions above, 
that is, for EQl and EQ3-EQ6. For EQ2, a qualitative analysis was employed, which involved 
coding provider responses to a query asking for descriptions of the impediments they encoun- 
tered that inhibited delivery of sessions for which no reports were received. Significance test- 
ing (paired t tests) was used to determine statistical significance of differences between 
participants’ assessments of their knowledge, practice, and attitudes/beliefs before (pre) and 
after (post) the professional development they attended (EQ7). Practical significance was 
studied using the Cohen’s d statistic to determine the effect size of those pre and post measures 
(EQ7). Also, we reviewed policy and state code requirements for various types of collabora- 
tions among the four major groups in the PD Master Plan to put findings into context. 

Results 

During the course of the 2013-2014 reporting year, 1,056 provider reports were com- 
pleted, and 4,758 usable participant survey responses were received from a sample of 9,416, 
representing a 50.5% response rate, which was lower than in previous years, when rates 
tended to range from 63% to 66% (Table A 1, page 35). 2 This level of response to the partici- 
pant survey poses challenges with regard to reporting on some measures of effectiveness and 
impact for individual providers— especially those that submitted relatively few participant e- 
mail addresses— as confidence intervals are larger than 5% for 10 of the 21 providers, even at 
a 90% confidence level. For this reason, caution should be used in interpreting findings for 
individual providers, and this report will focus less than usual on individual provider results 
from the participant survey. At the provider group level, however, the margins of error fall 
within a generally acceptable range (±1% to ±5%) for all four groups at the 90% confidence 
level; at the 95% confidence level results for three of the groups ranged from ±2% to ±3%, with 
institutions of higher education having a larger margin of error (±6%) (Table A 1, page 35). 


2 The response rate is slightly higher, that is 54%, if calculating it from the random sample of 
viable e-mail addresses (8,834) only. 
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EQl. What was the level of implementation for the PD Master Plan, overall and by 
individual providers? 

As one measure of the effectiveness of the plan’s implementation, this question will be 
addressed in two parts, including the overall participation in developing and implementing 
the master plan including the percent of professional development offerings that were planned 
versus the ones delivered; and the level of participation in the evaluation of the PD Master 
Plan. 



Figure 1. Trends in Participation of Provider Groups in the PD 

Master Plan, 2011-2012 through 2013-2014 

Data Source: 2013-2014 PD Master Plan Session Report database 


The number of providers re- 
mained about the same as dur- 
ing 2012-2013, with one 
exception. One more institution 
of higher education (IHE), Fair- 
mont State University, joined 
Marshall University to partici- 
pate in the plan. Eight other 
IHEs are required by W. Va. 
Code (§18-21) to participate, 
but were not successfully re- 
cruited by WVDE to do so. 

Decentralization of the de- 
livery of professional develop- 
ment, as called for in education 
reform legislation passed in 
2013, appears to be working 
(Figure 1). While the overall 
number of session topics sub- 
mitted for the plan has risen 
strongly from 218 in 2011-2012, 
to 408 the following year, to 
479 in 2013-2014, the number 
submitted by each of the four 
groups has shifted dramatically 
since 2012-2013, with the 
WVDE declining in both num- 
ber of topics and number of 
participants, and RESAs in- 
creasing. Yet, the WVDE re- 
mains the provider group with 
the most topics and most par- 
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ticipants. The rise in RESA topics and participation may reflect, in part, a decision by the di- 
rectors to participate more fully in the PD Master Plan than they have in previous years, in 
addition to true increases in the professional development they provide. In previous years, 
RESAs’ annual reports reflected far more activity than that indicated by their reporting for the 
PD Master Plan. In 2013-2014, however, RESAs received additional funding to bolster their 
staffing for professional development and were assigned more responsibility for it than was 
previously assigned to the WVDE. The dramatic increases in the number of topics RESAs sub- 
mitted to the plan, from three topics per RESA (one aligned with each of the Board goals) to 
as many as 49 topics (RESA 1), likely did reflect to some extent increases in their professional 
development activities, as well as increased participation in the PD Master Plan. 

Most professional development planned by providers in the PD Master Plan was later 
delivered and reported. In all, 479 session topics were approved for the PD Master Plan for 
2013-2014; of those over 80% (384) were reported as delivered— a similar percent as in pre- 
vious years. About half of the organizations in the PD Master Plan provided over 90% of what 
they had planned. Three providers delivered and reported less than half of what was in their 
plan, including the WVDE Office of Secondary Learning (47.6%), the WVDE Office of Career 
and Technical Accountability (44.4%), and Marshall University's Clinical Experiences and 
Professional Development Schools (33.3%; see Figure 2 below and Table A 2, page 36). 

Implementation of the evaluation of the PD Master Plan 

Participation in the PD Master Plan also includes participation in its evaluation, which 
is done through session reports submitted online via a SurveyMonkey instrument. Providers 
may offer a topic listed for them in the PD Master Plan once or multiple times with different 
groups of individuals. Once a session is completed with a particular group of individuals in- 
cluding any follow-up, the provider submits a session report, which includes email addresses 
of participants. The submission of email addresses is central to the ability of this evaluation to 
contact participants and collect their impressions of their professional learning experiences 
via the Participants Survey. 

Providers varied considerably in their compliance with this aspect of participation in 
the PD Master Plan evaluation. Of the four major groups, the Center for Professional Devel- 
opment and regional education service agencies performed best, supplying 98% and 94% of 
their participants’ email addresses, respectively. Institutions of higher education supplied 
about three quarters (73%) and the West Virginia Department of Education supplied about 
two thirds (68%). Four offices in the Department were primarily responsibility for the rela- 
tively low rate of compliance with the e-mail address requirement. The Office of Professional 
Preparation (Certification) submitted only 27% of the email addresses for participants in their 
sessions; similarly the Office of Career and Technical Accountability and Support submitted 
23%. The Office of Secondary Learning provided only 4% of the requisite email addresses 
while the Office of Institutional Education Programs provided none. 
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Total (planned 479) 
CPD (planned 65) 
RESAs (planned 156) 
WVDE (planned 233) 
IHEs (Planned 25) 



WVDE - Office of Special Programs (11 
WVDE - Office of School Improvement (6 
WVDE - Office of Federal Programs (1 
WVDE - Office of Early Learning (7 
RESA8 (2 
RESA7 (16 
RESA6 (23 
RESA5 (21 
Fairmont State University (4 
Center for Professional Development (65 
WVDE - Office of Assessment (12 
RESA2 (9 
RESA3 (24 

WVDE - Office of Instructional Technology (81 
WVDE - Office of Career and Tech. Instr. (60 

RESA4 (12 
RESA1 (49 

WVDE - Office of Prof. Prep. (Certification) (13 
WVDE - Office of Institutional Educ. Prog. (12 
Marshall University - June Harless Center (15 
WVDE - Office of Secondary Learning (21 
WVDE - Office of Career and Tech. Account. y (9 
Marshall University - Clinical Exp & PDS (6 


planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 

planned) 


1 100.0 
1 100.0 
1 100.0 
1 100.0 
1 100.0 
1 100.0 
1 100.0 
1 100.0 
1 100.0 


I 93.8 
91.7 


■ 88.9 
83.3 


I 77.8 
75.0 
75.0 


I 69.4 
I 69.2 
66.7 


60.0 


I 47.6 
44.4 


0.0 20.0 40.0 60.0 80.0 100.0 

Percent of planned session topics reported as delivered 


Figure 2. Percentage of Sessions Included in the PD Master Plan That Were Reported as Delivered 
by Provider Group and Provider 

Data Source: 2013-2014 PD Master Plan Session Report database 


EQ2. What were the impediments, if any, to the full implementation of the PD Master 
Plan? 


Each of the providers that had session titles in the plan for which no reports were received was 
offered the opportunity to provide explanations for why the sessions were not held or re- 
ported. For more than half of the topics not reported (49), providers explained that the session 
actually was or would be provided, but on a different schedule (just before or just after the 
reporting year), was delivered by other means or providers, or was provided but not reported. 
Other sessions were cancelled due to lack of interest (22) or to unforeseen circumstances (8). 
An additional 14 sessions were not reported as delivered with no explanation provided, or had 
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other reporting errors. For a breakdown by provider see Table A 4 (page 38). Other summary 
details are available in Table A 5 (page 39). 

EQ3. To what extent did providers' offerings reflect the WVBE Standards for Professional 
Learning? 

As mentioned earlier, the Board is required in code to adopt standards for professional 
learning, which it did in 2012. The Board standards may best be viewed as standards to be met 
by individual schools and districts rather than by providers because, taken together, they make 
up a system for continuous school improvement in which all members of a school community 
have roles to play and decisions about professional development are based on students and 
teachers learning needs (see Relevant Scholarship, page 2). In such a system, providers may 
play a role, however, in delivering training on a topic or skill set educators have identified as 
being needed. 

There was no practical way to learn from participants whether providers were aligning 
their offerings with all seven Board standards. We did, however, ask providers themselves 
which of the Board standards they had met for each of the sessions they reported. Results are 
shown in Table 2. While CPD claims to have met all standards for all sessions they offered, 
other providers were more circumspect in their assessments. Overall, the standards that pro- 
viders most often believed they had met were Standard 7 (“Aligns its outcomes with educator 
performance and student curriculum standards”) and Standard 5 (“Integrates theories, re- 
search, and models of human learning into learning designs to achieve its intended out- 
comes”). The two standards providers least often reported meeting were Standard 4 (“Uses a 
variety of sources and types of student, educator, and system data to plan, assess, and evaluate 
professional learning”) and Standard 6 (“Applies research on change and sustains support for 
implementation of professional learning for long-term change”). 


Table 2. Reported Number and Percent of Sessions That Met Each Board Standard by Provider 


Board standard met 

All (n 

= 1,320) 

CPD (n = 163) 

IHEs 1 

(n =45) 

RESAs (n =384) 

WVDE 

(n =428) 

N 

% 

N 

% 

N 

% 

N 

% 

N 

% 

1. Learning communities 

673 

66.0 

163 

100.0 

22 

48.9 

305 

79.4 

183 


42.8 

2. Leadership 

703 

68.9 

163 

100.0 

18 

40.0 

297 

77.3 

225 


52.6 

3. Resources 

669 

65.6 

163 

100.0 

25 

55.6 

291 

75.8 

190 

■ 

44.4 

4. Data 

575 

H^6.4 

163 

100.0 

19 

42.2 

270 

70.3 

123 


28.7 

5. Learning designs 

733 

71.9 

163 

100.0 

30 

66.7 

316 

82 3 

224 


52.3 

6. Implementation 

600 

^■8.8 

163 

100.0 

17 

37.8 

292 

76.0 

128 


29.9 

7. Outcomes 

796 

78.0 

163 

100.0 

42 

93.3 

308 

80.2 

283 

66.1 


Data Source: 2013-2014 PD Master Plan Session Report database 


EQ4. To what extent did providers' offerings reflect research-based practices? 

The final measure of effectiveness used an index based on findings from a review of 
the research literature on effective professional development practices (Hammer, 2014; see 
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Appendix A, page 33). The index was more 
focused on what providers can do— versus 
what school and district staff can do— to de- 
sign and implement professional develop- 
ment experiences that have a collection of 
qualities shown to be associated with 
changes in teacher practice and improved 
student performance. The index is, there- 
fore, well suited to eliciting participant re- 
ports about which of these qualities were 
present in the sessions they were asked to 
comment upon (see box for items in in the 
index). Response options included 1 
(strongly disagree), 2 (disagree), 3 (agree), 
4 (strongly agree), and not applicable. The 
index scores were based on the percentage 
of the 15 items that participants agreed or 
strongly agreed were present in the session 
they attended. 

The index did have limited use for 
some role groups included in the survey 
sample, however, including RESA staff, 
district central office staff, and a miscella- 
neous other category. Research on effective 
approaches to professional development is 
much scarcer for these role groups. The in- 
dex, therefore, is based on the richer re- 
search base focused on classrooms and 
schools. Table A 6 (page 40) shows the per- 
centages of respondents from each role 
group that agreed or disagreed with each 
item or considered it not applicable. Sub- 
stantial percentages of the non-school- 
based respondents found most of the items 
in this portion of the survey not applicable. 
These data are also summarized in Figure 
A 1 (page 41), which shows that, overall, ed- 
ucators in the classroom were most likely to 
find the items in the Research-Based PD 
Practices Index applicable (whether or not 
they agreed), non-school personnel were 
least likely, with principals bridging the 

gap- 


RESEARCH-BASED PD PRACTICES INDEX 

The professional development . . . 

CONTENT FOCUS 

1. Deepened my knowledge of the content area it 
covered. 

2. Strengthened my instructional approaches for 
teaching the content area it covered. 

3. Used curriculum materials I will be using with my 
students. 

COHERENCE 

4. Was relevant to reaching my school or district's 
goals for student learning. 

5. Was challenging and helped me develop my skills 
to a new level. 

6. Spent too much time repeating concepts I have 
learned before, (reverse scale item) 

ACTIVE LEARNING 

7. Included opportunities for discussions, reviewing 
student work, and/or written exercises. 

8. Included valuable time to plan for implementa- 
tion in my classroom, school, or district. 

9. Allowed me opportunities to practice what I was 
learning and receive constructive feedback. 

COLLECTIVE PARTICIPATION 

10. Included colleagues in my content area, grade, or 
specialization from my school or district. 

11. Motivated my colleagues and me to collaborate 
more in our shared work with students. 

12. Helped my colleagues and me arrive at a common 
understanding and approach to instruction. 

DURATION AND TIMESPAN 

13. Had enough contact hours to help me learn the 
content and skills it encouraged. 

14. Offered enough experiences during the school 
year for me to develop and successfully apply 
new skills. 

15. Required more of my time than I think was 
needed for this topic, (reverse scale item) 
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Therefore, the following analyses will focus on the school-based respondents: princi- 
pals/ assistant principals, general and special education teachers, library media specialists, in- 
structional support teachers, and paraprofessional/aides. Other sections of the survey will 
allow us to examine the views of the non-school-based respondents. 

Looking only at school-based respondents, the level of agreement varied both by role 
group and by research-based practices (Figure 3). Overall, there was the greatest agreement 
that the professional development had focused on a content area, including instructional prac- 
tices and use of cur- 
riculum materials; 
principals, as might 
be expected, were 
the exception. Ac- 
tive learning scored 
lowest, followed by 
respondents' esti- 
mation that the ses- 
sion have provided 
sufficient duration 
and timespan to al- 
low them to apply 
what they were 
learning. Partici- 
pants were largely in 
agreement that the 
professional devel- 
opment had been 
well aligned (coher- 
ent) with their own 
needs and those of 
the school and dis- 
trict. 

Looking once 
again only at school- 
based respondents, 
but this time focus- 
ing on differences 
among provider 
groups, active learn- 
ing was the re- 
search-based PD 

practice least often present, followed by duration/ timespan with, in both cases, RESAs scoring 
the lowest (Figure 4). Content focus was the highest followed closely by coherence with WVDE 
scoring highest for the former and IHEs for the latter. About three quarters of participants in 
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Figure 3. Percent of School-Based Respondents That Agreed or Strongly Agreed 
That Five Research-Based PD Practices Were Present in the Sessions They Attended 
by Role Group 

Data Source: School-based respondents to WVBE PD Master Plan Participant Survey 
2013-2014 
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professional learning offered by the Center for Professional Development, institutions of 
higher education, and the West Virginia Department of education indicated that the training 
they received was of sufficient duration and timespan for the content it covered. About two 
thirds of participants in RESA trainings reported the same. 

Looking at individual providers (Table A 7 page 42), the top three overall performers 
in the use of research-based professional development were the WVDE Office of Career and 
Technical Accountability and Support with respondents agreeing or strongly agreeing that the 
15 research-based professional development practices were present 89.6% of the time, Mar- 
shall University June Harless Center (85.6%), and the WVDE Office of Instructional Technol- 
ogy (85.2%). The three lowest performers were RESA 7 (75.1%), RESA 8 (74.9%), and the 
WVDE Office of Professional Preparation (Certification) (74.9%). The overall percentage of 
respondents who agreed/strongly agreed that the session they attended used the research- 
based professional development practices was 82.2%. 


Content focus 


Coherent 


Q_ 

Q 

CL 

~o 

O) 

CO 

03 

_Q 


Active learning 


£ Collective participation 


Duration/timespan 


80.0 
| 82.9 
80.3 

87.0 


■ 81.4 

■ 83.2 
111 

81.8 


71.7 
I 76.3 


65.9 


75.3 


73.6 


83.2 

80.1 


75.1 


67.7 


74.9 

74.8 

7 

75.0 


0.0 20.0 40.0 60.0 80.0 100.0 

Percent agreed/strongly agreed 


■ CPD (n = 638) ■ IHEs (n = 111) ■ RESAs (n = 1,615) ■ WVDE (n = 1,515) 


Figure 4. Percent of School-Based Respondents That Agreed or Strongly Agreed That 
Five Research-Based PD Practices Were Present in the Sessions They Attended by 
Provider Group 

Data Source: School-based respondents to WVBE PD Master Plan 
Participant Survey 2013-2014 
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As mentioned earlier, the index used in the analyses described above was made up of 
15 items, three for each of the five research-based professional development practices. A more 
granular look at the survey results for school-based respondents— that is, examining the re- 
sults for each of the items in the Research-Based PD Practices Index— gives a few more clues 
about what each of the individual providers may want to look at in their own planning and 
conduct of professional learning experiences (see Table 3). 

Table 3. Percent Agreed/Strongly Agreed the Session They Attended Met This Criteria by Provider 

Percent agreed/strongly agreed with this item 


Professional development included . . . 

CPD 
(n = 638) 

IHEs 
(n = 111) 

RESAs 
(n = 1,612) 

WVDE 
(n = 1,507) 

Content focus 

Focus on content knowledge 

91.4 

90.1 

89.2 

92.3 


Focus on content instruction 

80.9 

82.0 

80.7 

86.9 


Use of curriculum materials 

67.7 

76.6 

71.0 

81.9 

Coherence 

Alignment with school/district goals 

87.0 

88.3 

88.7 

91.0 


Alignment with own training needs 

86.1 

84.7 

77.6 

84.5 


Not a repeat of material covered before 

71.2 

76.6 

66.7 

69.8 

Active learning 

Discussion/review of student work 

81.0 

79.3 

76.3 

83.2 


Planning for implementation 

60.8 

70.3 

54.7 

65.9 


Practice and feedback 

73.2 

79.3 

66.7 

77.0 

Collective 

Participation with colleagues from own school 

80.7 

90.1 

88.9 

78.3 

participation 

Motivation for collaboration 

68.7 

79.3 

74.0 

72.2 


Development of common understandings 

71.5 

80.2 

77.3 

74.7 

Duration & 

Enough contact hours 

83.2 

78.4 

71.3 

83.5 

timespan 

Enough experiences during school year 

67.7 

71.2 

60.8 

73.5 


Not too much time for topic covered 

73.7 

74.8 

71.1 

68.0 


Data Source: School-based respondents to WVBE PD Master Plan Participant Survey 2013-2014 


The following are the highest- and lowest-scoring items in the survey among school-based 

respondents: 

Center for Professional Development 

• Highest scores— Focus on content knowledge, Alignment with school/district goals, 
and Alignment with own training needs 

• Lowest scores— Planning for implementation, Enough experiences during school year, 
and Use of curriculum materials 

Institutions of higher education (Fairmont University and Marshall University) 

• Highest scores— Focus on content knowledge, Participation with colleagues from own 
school, and Alignment with school/ district goals 

• Lowest scores— Planning for implementation, Enough experiences during school year, 
and Right amount of time for topic 

Regional education service agencies 

• Highest scores— Focus on content knowledge, Participation with colleagues from own 
school/ district, and Alignment with school/district goals. 

• Lowest scores— Planning for implementation, Enough experiences during the school 
year, and Did not repeat material seen before 

West Virginia Department of Education 
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• Highest scores— Focus on content knowledge, Alignment with school district goals, 
and Focus on content instruction 

• Lowest scores— Planning for implementation, Right amount of time for topic, and Did 
not repeat material seen before 


For one of the five research-based professional development practices, dura- 
tion/timespan, we also collected data from the providers. As part of their session reporting, 
providers indicated the duration of each session in hours and the beginning and ending dates, 
from which we were able to ascertain the timespan in days. As mentioned earlier, research has 
shown that at least 30 hours of professional development are needed to affect teacher practice 
and improve student learning. Additionally, when professional learning takes place over an 
extended period of time, there are more opportunities for participants to practice new skills, 
and follow-up with trainers and fellow participants as they encounter new challenges. 


Looking first at du- 
ration, the largest por- 
tion (37.1%) of the 
participants attended 
technical training last- 
ing a half day up to 2 
days; the second most 
common duration cate- 
gory (27.6%) was infor- 
mational sessions, last- 
ing up to 4 hours. Only 
about 18% of attendees 
were in sessions lasting 
the recommended 30 
hours or more. 

The different pro- 
vider groups varied 
considerably. About 
30% of attendees at 
CPD-led sessions and 
about 27% at WVDE- 
led sessions received 
30 or more contact 
hours. Less than 1 per- 
cent of attendees at 
RESA-led sessions re- 
ceived that number of 


Duration Category 


All (n = 33,196) 
CPD(n = 3,785) 
IHEs (n = 762) 
RESAs (n = 11,289) 
WVDE (n = 17,360) 


27.6 

37.1 

17.0 

18.3 
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29.6 
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18.4 

21.1 
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Timespan Category 


All (n = 33,196) 
CPD (0 = 3,785) 
IHEs (n = 762) 
RESAs (n = 11,289) 
WVDE (n = 17,360) 


50.6 


22.9 


5.2 13.0 8.3 


48.8 


35.0 


16.2 


36.5 


18.2 


9.8 


35.4 


74.1 
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Figure 5. Distribution of Attendees Across Duration and Timespan Categories 
by Provider Group 

Data Source: 2013-2014 PD Master Plan Session Report database 


contact hours, which serves to confirm the comparatively low ratings given to RESAs by re- 
spondents to the participant survey for duration and timespan (see Figure 4, page 16 and Table 
3, page 17). There was tremendous variation among RESAs, however. For example, about 80% 
of attendees in RESA 5 sessions and more than half of those in RESA 6 sessions received 14 or 
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more contact hours. There was equal variation among WVDE providers. Nearly 80% of at- 
tendees at WVDE Office of Instructional Technology received the recommended level of at 
least 30 hours of training, while nearly all (92%) of the attendees at sessions provided by the 
Office of Institutional Education Programs received 4 or fewer contact hours. For results by 
individual provider, see Figure A 2 (page 43). 

Turning now to timespan, measured here in the number of days that elapsed between the 
beginning date and the ending date of sessions, there is again considerable variation (Figure 
5, page 18). Overall, slightly more than half of all participants attended sessions that began 
and ended on the same day. Institutions of higher education tended to have the greatest at- 
tendance at sessions with longer timespans, with about 45% of their participants attending 
sessions taking place over at least a calendar month including more than a third attending 
sessions that spanned more than 60 days. About three quarters of RESA participants attended 
training that began and ended on the same day. Among RESAs, RESAs 5 and 6 had training 
with the longest timespans. More than 80% of RESA 5 attendees and more than 70% of those 
for RESA 6 were engaged in training that spanned at least 2 days; both RESAs had at least 
some participants who were engaged in training that spanned more than 60 days. All of the 
training offered by two WVDE offices (Office of Federal Programs and Office of Institutional 
Education Programs) began and ended the same day. On the other hand, nearly 80% of at- 
tendees in the Office of Instructional Technology were engaged in training that spanned at 
least 31 days. 

Efficiency 

EQ5. To what extent did providers collaborate in the delivery of professional 
development? 

One of the purposes of the PD Master Plan is to increase the efficiency of the statewide 
system of professional development by reducing duplication of effort. We studied the issue of 
efficiency by looking at the number of collaborative partnerships providers in the PD Master 
Plan engaged in to conduct their sessions, assuming that collaboration is an antidote to siloed, 
independent offerings. In this analysis we focused only on collaborations with other providers 
in the PD Master Plan. Partnerships with vendors, consultants, counties, and other state agen- 
cies are not included in this analysis. 

Figure 6 shows, on average, the number of collaborators state providers had for each 
of their sessions. The providers who exercised the greatest degree of collaboration were lo- 
cated in the Department of Education with RESAs coming in second. Looking at individual 
providers, the top six were in the WVDE. Top collaborators were the Offices of Early Learning, 
Special Programs, Instructional Technology, Professional Preparation, Assessment and Ac- 
countability, and Institutional Education Programs. A major provider of professional devel- 
opment, the Center for Professional Development worked almost completely independently 
of other providers according to their session reports. Figure 7 provides another look at the 
nature of the collaborations for each provider group by showing with which other providers 
each group collaborated most frequently. RESAs and WVDE offices were the most prevalent 
collaborators with each other. 
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All providers (n = 1,069) 

West Virginia Department of Education (n = 449) 
Regional education service agencies (n = 406) 
Institutions of higher education (n = 45) 
Center for Professional Development (n = 169) 

WVDE Office of Early Learning (n = 8) 
WVDE Office of Special Programs (n = 86) 
WVDE Office of Instructional Technology (n = 213) 

WVDE Office of Professional Preparation (n = 26) 
WVDE Office of Assessment and Accountability (n = 27) 
WVDE Office of Institutional Education Programs (n = 8) 

RESA 4 (n = 73) 
RESA 1 (n = 113) 
WVDE Office of Secondary Learning (n = 15) 
RESA 8 (n = 37) 

Marshall University--June Harless Center (n = 35) 
WVDE Office of School Improvement (n = 9) 
WVDE Office of CTE Instruction (n = 50) 
RESA 5 (n = 26) 
RESA 7 (n = 23) 
RESA 3 (n = 70) 
RESA 6 (n = 37) 

Center for Professional Development (n = 169) 
WVDE Office of Federal Programs (n = 2) 
WVDE Office of CTE Accountability and Support (n = 5) 

RESA 2 (n = 27) 

Marshall University-Clinical Exp. & PD Schools (n = 4) 
Fairmont State University (n = 6) 
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Figure 6. Average Number of Collaborations per Session by Provider 

This graph shows the overall rate of collaboration per session (top bar), the rates for each of the provider 
groups (next four bars down), and the rates for each individual provider (remaining bars). Data source: 
2013-2014 PD Master Plan Session Report database 
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Figure 7. Provider Group Collaborations 
2013-2014 PD Master Plan Session Report database 
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Impact 

Impact will be examined by addressing the final two evaluation questions related to 
coverage of board goals and participants’ estimation of how their own knowledge, practice, 
and attitudes/beliefs were affected by the professional development session we asked them to 
respond to. 

EQ6. How well did providers' offerings address the WVBE's 2013-2014 Goals for 
Professional Learning? 

We will examine two dimensions to this question— how well the goals were covered by 
the providers, and participants’ views about how helpful the professional development was in 
meeting the Board’s goals. 

Proportion of PD offerings targeting each of the goals overall 

About 80% of sessions (n = 858) were focused on Goal 2 with 73% of attendees (n = 
24,233). Goal 2 sessions were among the longest in duration, second only to sessions focused 
on Goal 1. By comparison, sessions focused on Goals 3 and 4 had an average duration of less 
than half that of Goals 1 and 2 (Table 4). 

The longer average duration for Goal 1 sessions was attributable to the 15 sessions of- 
fered by the WVDE Office of Instructional Technology, which were attended by more than 
1,200 individuals and had 45 contact hours each. The sessions were offered to help attendees 
obtain credentials needed to serve in the state’s prekindergarten classrooms. 


Table 4. Coverage of Board Goals for Professional Learning 


Goal 


Sessions Attendance Duration 

(n) (n) (mean hours) 


All sessions 

Professional development for the 2013-2014 school year 


1,069 


33,194 


13.6 


2. 


r 3. 


4. 


Increase the knowledge and skills of all pre-K 
educators to deliver a comprehensive pre-K-third 
grade approach to early childhood education that 
includes a balanced approach to early literacy. 

Increase deep content knowledge and proficiency in 
designing and delivering standards-driven 
instruction and assessments for all preK-12 West 
Virginia educators. 

Improve leadership competencies for principals 
and assistant principals in order to support high 
quality teaching and learning. 

Support the full implementation of the revised 
educator evaluation system. 



Data source: 2013-2014 PD Master Plan Session Report database 
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The relatively high average duration for Goal 2-focused sessions was attributable to six 
providers, all of which had mean durations above the grand mean shown in Table 4. In de- 
scending order by average duration they were WVDE Office of Instructional Technology (28.9 
hours, 4,102 attendees; RESA5 (21.8 hours, 912 attendees); Marshall University June Harless 
Center (19.9 hours, 460 attendees); WVDE Office of Early Learning (17.0 hours, 270 at- 
tendees); Center for Professional Development (16.6 hours, 2,268 attendees); and RESA 6 
(14.8 hour, 845 attendees). 

Participant reports of PD' s helpfulness in meeting board goals 

Participants’ were asked in four items if the session they had attended was helpful in 
meeting each of the four Board Goals for Professional Learning. Response options included 1 
(not applicable), 2 (strongly disagree), 3 (disagree), 4 (agree), and 5 (strongly agree). The data 
in Figure 8 reflect only the responses of participants in sessions providers indicated were 
aligned with particular goals. So for example, if a participant attended a session that a provider 
said was aligned with Goal 1, only his/her response to the Goal 1 item was included in the 
analysis and his/her responses to the items for Goals 2-4 were ignored. Although ordinarily 
we would not include responses of “not applicable” when calculating percentages, we thought 
such a response to a session that providers’ considered aligned to a particular goal was a form 
of nonagreement that the session had been helpful in moving them toward that goal. Figure 8 


shows a high level of general agreement— at least 75%— that the sessions they attended had 
been helpful. 

Goal 1 . Comprehensive Early Literacy (n = 374 ) 

Goal 2 . Content and Pedagogy to Standards (n = 3 , 063 ) 

Goal 3 . Leadership Competencies (n = 561 ) 

Goal 4 . Educator Evaluation System (n = 414 ) 

0 . 





.0 20.0 40.0 60.0 80.0 100.0 

Percent of respondents who agreed/strongly agreed 


Figure 8. Percent of Respondents Who Agreed or Disagreed That the Session Was Helpful in Meeting the 
Targeted Board Goal 


EQ7. What was the impact of the professional development offered through the 2013- 
2014 PD Master Plan on educators' knowledge, practice, and attitudes? 

We used a retrospective pretest/posttest design to assess the extent to which survey 
respondents perceived a change in their own knowledge, behaviors/practice, and beliefs/atti- 
tudes as a result of participating in professional development. The survey contained three 
pairs of items that asked respondents to use a 4-point Likert-type scale (o [not at all], 1 [to a 
small extent], 2 [to a moderate extent], 3 [to a great extent]), to rate the extent to which they 
agreed with statements about themselves both before and after having participated in the pro- 
fessional development session they attended. A fifth response category was included, but only 
used to allow respondents to indicate the item was not applicable to them. These responses 
were not used when calculating mean scores. 
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Pair l. Before participating in this PD, to what extent were you knowledgeable about the topic 
it covered? 

After participating in this PD, to what extent are you knowledgeable about the topic it 
covered? 

Pair 2. Before participating in this PD, to what extent did you practice behaviors or skills it 
taught? 

After participating in this PD, to what extent do you practice behaviors or skills it 
taught? 

Pair 3. Before participating in this PD, to what extent did you hold attitudes/beliefs it encour- 
aged? 

After participating in this PD, to what extent do you hold attitudes/beliefs it encour- 
aged? 

Aggregated presession scores averaged between 1.7 and 2.0, indicating that partici- 
pants, overall, thought they had a moderate or slightly less than moderate level of knowledge, 
skill, and attitude/belief prior to engaging in the session. They assessed themselves just below 
the midpoint between the moderate and great levels after the session, indicating that partici- 
pants, overall, thought they had grown professionally as a result of the experience. 


Table 5. Overall Average Self-Scores for Extent of 
Knowledge, Practice, and Beliefs Before 
and After Professional Development 
Average score 

Before PD After PD 


Knowledge about topic (n = 

1.7 

2.3 

4,523) 



Practice of behaviors or 

1.8 

2.3 

skills (n = 4,312) 



Held attitudes and beliefs 

2.0 

2.4 


(n = 4,362) 

Note: 0 = not at all, 1 = to a small extent, 2 = to a 
moderate extent, 3 = to a great extent; responses 
marked not applicable were excluded from the 
analysis. 

Source: WVBE PD Master Plan Participant Survey 2013- 
2014 


To test the statistical significance 
of these findings, we ran a series of paired- 
samples t tests using respondents’ pre- and 
post-ratings. When statistically significant 
differences were found (i.e., p <.05), it is 
reasonable to say that the differences ob- 
served between participants’ pre- and 
posttest results are not likely due to 
chance. That is, there is some systematic 
reason underlying the difference. This 
analysis, however, does not allow one to 
infer a cause for the difference. It merely 
describes the presence of a significant dif- 
ference. 


Significance testing revealed that 
the results were significant at the p <.05 level for all but three of the 78 tests we ran— and the 
great majority of those tests were statistically significant at the p < .001 level (see Table A 9, 
page 46 in Appendix B). 
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One limitation of significance testing is that it tells us very little about the magnitude 
of any observed differences. We detect a difference, but cannot tell from the t test if the differ- 
ence is meaningful in a practical sense. Calculating an effect size is one way to explain the 
magnitude of any statistically significant differences. In this study, we used Cohen’s d as a 
measure of effect size. This statistic is often used in simple retrospective pretest/posttest de- 
signs, although its interpretation is often debated in social sciences (see the Limitations of the 
Study section, page 26, for more about this debate). The guidelines we used for interpreting 

the meaning of the effect sizes in this study are 
Table 6. Interpretation of Effect Size found in Table 6. 

Estimates Used in this Study 


Value for Cohen's d 

Interpretation 

Less than .4 

Small effect 

.4 to .7 

Moderate effect 

.8 or 1.1 

Large effect 

1.2 and above 

Very large effect 


Aggregating all results, respondents per- 
ceived a large impact on the extent of their 
knowledge as a result of attending the session, with 
moderate effects on their practice and atti- 
tudes/beliefs (Figure 9). This pattern held across 
the provider groups with a couple of exceptions: Institutions of higher education saw very 
large effects for knowledge compared with the large effects seen by other groups, and the 
Center for Professional Development saw large effects for practices compared with the mod- 
erate effects for the other groups. 


Figure 10 displays the range of ef- 
fects for individual providers, which 
generally follow the pattern described 
above, with larger effects for 
knowledge, more moderate effects for 
practice, and the smallest effects for 
attitudes and beliefs. Some notable 
results included the following: 

• The Center for Professional Devel- 
opment saw large effects on prac- 
tice. 

• Among the institutions of higher 
education, participants in profes- 
sional development provided by 
Marshall University reported the 
greatest growth compared with 
Fairmont University. 

• Among RESAs, RESA 6 was the 
only one that registered very large 
effects on knowledge and large ef- 
fects on practice. 

Figure 9. Reported Impact of PD on Knowledge, Practice, 

and Attitudes by Provider Group 

See Table 6 for interpretation of these effect sizes. 

Data source: WVBE PD Master Plan Participant Survey 
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sional Preparation (Certification), OSI = School Improvement, 
OSL = Secondary Learning, OSP = Special Programs 


Figure 10. Perceived Impact of Professional Development (Retrospective Pre/Post): Effect Size by Provider 
The following values are generally assigned to effect sizes: Less than 0.4 = small effect; 0.4 to 0.7 = moderate 
effect, 0.8 to 1.1 = large effect, and 1.2 and above = very large effects. 

* T test results for impacts on beliefs (only) were not statistically significant. 
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• Among WVDE offices, Career and Technical Accountability and Support, Federal Pro- 
grams, and Instructional Technology all saw very large effects on knowledge. Career and 
Technical Accountability and Support also saw large effects on practice, as did Special 
Programs. 

Discussion 

The PD Master Plan included more professional development sessions than ever be- 
fore, giving us an opportunity to measure the effectiveness, efficiency, and impact of profes- 
sional development by major providers (2014-2015 includes even more). Major findings for 
2013-2014 include the following, arranged here by effectiveness, efficiency, and impact: 

Effectiveness of the Master Plan 

• The PD Master Plan included more topics in 2013-2014 (479) than the previous year 
(408) and at 33,196 participants, an increase of about a thousand educators in attend- 
ance. 

• Regarding the effectiveness of providers in meeting the Board Standards for Profes- 
sional Learning, overall, the standards that providers most often believed they had met 
were Standard 7 (“Aligns its outcomes with educator performance and student curric- 
ulum standards”) and Standard 5 (“Integrates theories, research, and models of hu- 
man learning into learning designs to achieve its intended outcomes”). The two 
standards providers least often reported meeting were Standard 4 (“Uses a variety of 
sources and types of student, educator, and system data to plan, assess, and evaluate 
professional learning”) and Standard 6 (“Applies research on change and sustains sup- 
port for implementation of professional learning for long-term change”). 

• Regarding the extent to which providers’ offerings reflected five research-based pro- 
fessional development practices, overall, there was the greatest agreement that the 
professional development had focused on content and content pedagogy. Active 
learning scored lowest; only slightly higher was respondents' estimation that the ses- 
sion had provided sufficient duration and timespan to allow them to apply what they 
were learning. Participants were largely in agreement that the professional develop- 
ment had been well aligned ( coherent ) with their own needs and those of the school 
and district. 

• Providers’ session reports seemed to confirm the findings about duration and 
timespan above. Of the 33,000 attendees reported by providers, about 5,700 educators 
participated in professional development lasting at least 30 hours; another 5,400 had 
from 14 to 29 contact hours— durations shown by research to be the minimum needed 
to change teacher practice and impact student learning. The remaining two-thirds of 
participants attended sessions ranging from 1 to 13 hours. 
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Efficiency of the Master Plan 

• The Legislature's call for decentralization of professional development seemed to be 
reflected in the trends for the four major providers in the Plan, with the WVDE de- 
creasing their number of offerings and participants from the previous year, while the 
RESAs' slate of offerings rose dramatically, as did their participant counts. CPD con- 
tinued its trend upward for both topics and participants, while IHES remained stable. 
Still, the WVDE continued to report the greatest number of both sessions held and 
participants in attendance. 

• RESAs and WVDE offices operated very collaboratively, partnering with each other 
and IHEs. According to CPD session reports, they worked almost completely inde- 
pendently of other providers. It should be noted, however, that CPD, like the RESAs, 
seeks input from the WVDE and others when setting its slate of offerings. 

Impact of the Master Plan 

• While all four of the Board’s Goals for Professional Learning received coverage, Goal 
2 (“Increase deep content knowledge and proficiency in designing and delivering 
standards-driven instruction and assessments”) was the focus of about 80% of ses- 
sions (n = 858) with 73% of attendees (n = 24,233). 

• The participant surveys showed a high level of general agreement— at least 75%— that 
the sessions they attended had been helpful in meeting the Board goal with which it 
was aligned. 

• According to self-reports, the professional development had large effects on educators’ 
knowledge of the PD topic, and moderate effects on their practice and attitudes/be- 
liefs. 


Limitations of the Study 

The response rate for the participant survey, 50.5%, was lower than in previous years, 
when rates tended to range from 63% to 66%. This result may have been due to at least three 
factors: (a) the schedule of the survey, which ran at least a month later than previous years (in 
June rather than May) due to the change in the reporting year called for in the PD Master Plan 
and to the lateness of some of the providers in submitting the session reports; (b) competition 
with two other statewide surveys about professional development, also from the Office of Re- 
search, that ran in close proximity to the PD Master Plan survey; (c) weariness with the PD 
Master Plan survey itself. 

The participant survey conducted in November-December 2013 and May-June 2014 
asked respondents to recall PD sessions they had participated in at some point in the past. In 
some cases, the sessions had taken place up to five months prior to the survey. For this reason, 
there is a possibility of temporal bias in survey participants’ responses. 

The use of a retrospective pretest/posttest methodology to assess changes in 
knowledge, behavior and skills, and attitudes and beliefs poses some concerns. We used this 
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methodology primarily because some researchers have argued that a phenomenon called re- 
sponse shift bias can occur when conducting traditional pretest/posttest designs. Response- 
shift bias “occurs when a participant uses a different internal understanding of the construct 
being measured to complete the pretest and posttest” (Moore & Tananis, 2009, p. 190). Con- 
sider this in context of professional development. Some respondents begin their involvement 
in professional development with a misconception that they are already well-versed in the 
content to be covered. When given a pretest, they rate their own knowledge, behavior and 
skills, and attitudes and beliefs very positively. However, over the course of the professional 
development, as they develop a deeper understanding of the content being covered, they real- 
ize they did not know as much as they originally thought. As such, when presented with the 
posttest, their frame of reference has shifted and they could potentially rate their knowledge, 
behavior and skills, and attitudes and beliefs lower than they did on the pretest. This can lead 
to problems in analyzing the impact of the professional development. For this reason, some 
researchers advocate for using retrospective pretest/posttest designs as we did in this study. 

Despite this strength of the retrospective pretest/posttest design, a recent research 
study conducted by Nimon, Zigarmi, and Allen (2011) found that using traditional pre- 
test/posttest designs leads to less biased estimates of program effectiveness. The authors pre- 
sent a compelling case that presenting both pre- and posttest items simultaneously on a single 
survey is among the most biased design options available to researchers and can significantly 
inflate effect size estimates. The authors recommend traditional pretest/posttest designs 
when possible and advocate for the implementation of a separate retrospective pretest to allow 
researchers to determine the presence of any response-shift bias. This design option, despite 
its strength, was not feasible in this study due to a mismatch between the scale of professional 
development offerings in the state and available evaluation staffing resources. Therefore, we 
recommend cautious interpretation of our own estimates of effect size, as they may be some- 
what inflated. 

Recommendations 

Based on these findings, we make the following recommendations. 

In keeping with the Board Standards for Professional Learning , 

• Increase the use of a variety of sources and types of student, educator, and system data 
to plan, assess, and evaluate professional learning, and 

• Increase the application of research on change and sustain support for implementation 
of professional learning for long-term change. 

With regard to the use of research-based PD practices, 

• Increase the use of active learning during professional development sessions, and 

• Provide sufficient duration (30 or more hours) and timespan (weeks or months) to 
allow participants opportunities to apply what they are learning. 

Based on factors present in the larger context of professional development in the state, we 
recommend that the West Virginia Board of Education and Department of Education 
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• Bring the last of the public IHEs with teacher education programs into the PD Master 
Plan (two participated in 2013-2014; five are participating in 2014-2015). 

• Promote the Board’s Standards for Professional Learning at the school and district 
level, so they will better guide educators’ planning. 

• Consider adoption of a model or standard for professional development providers that 
aligns with and supports local learning communities working to adopt the Board’s 
Standards for Professional Learning. 

• Revisit the purposes and possible uses of the PD Master Plan to guide professional 
development— aligning it to the Transforming Professional Development Initiative. 

• Consider ways to put the evaluation of the PD Master Plan to better use— reshaping 
the evaluation and its purposes as the plan itself is reshaped. 
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Appendix A. Research-Based PD Practices Index 


Research-Based PD Practices Index 


Please indicate to what extent you agree or disagree with the following statements about the professional 
development. 



The professional development . . . 

Not 

applicable 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

1 . 

Deepened my knowledge of the content area it 
covered. 

o 

o 

o 

o 

o 


Strengthened my instructional approaches for 
teaching the content area it covered. 

o 

o 

o 

o 

o 


Used curriculum materials 1 will be using with my 
students. 

o 

o 

o 

o 

o 

2. 

Was relevant to reaching my school or district's 
goals for student learning. 

o 

o 

o 

o 

o 


Was challenging and helped me develop my skills to 
a new level. 

o 

o 

o 

o 

o 


Spent too much time repeating concepts 1 have 
learned before. 

o 

o 

o 

o 

o 

3. 

Included opportunities for discussions, reviewing 
student work, and/or written exercises. 

o 

o 

o 

o 

o 


Included valuable time to plan for implementation in 
my classroom, school, or district. 

o 

o 

o 

o 

o 


Allowed me opportunities to practice what 1 was 
learning and receive constructive feedback. 

o 

o 

o 

o 

o 

4. 

Included colleagues in my content area, grade, or 
specialization from my school or district. 

o 

o 

o 

o 

o 


Motivated my colleagues and me to collaborate 
more in our shared work with students. 

o 

o 

o 

o 

o 


Helped my colleagues and me arrive at a common 
understanding and approach to instruction. 

o 

o 

o 

o 

o 

5. 

Had enough contact hours to help me learn the 
content and skills it encouraged. 

o 

o 

o 

o 

o 


Offered enough experiences during the school year 
for me to develop and successfully apply new skills. 

o 

o 

o 

o 

o 


Required more of my time than 1 think was needed 
for this topic. 

o 

o 

o 

o 

o 


Subscales include 


1. Content and content pedagogy 

2. Coherence 

3. Active learning 

4. Collective participation 

5. Duration and time span 
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Appendix B. Additional Data Tables and Figures 

Table A 1. Participant Survey Response Rates, Confidence Levels, and Confidence Intervals by Provider 


Group and Provider 



Sampling 

Random 






frame of 

sample of 



95% 

90% 


unique 

unique 

Usable 


confidenc 

confidenc 


email 

email 

responses 

Response 

e level, ± 

e level, ± 

Provider 

addresses 

addresses 

received 

rate 

% (Cl) 

% (Cl) 

Total 

12,860 

9,416 

4,758 

50.5 

1 

1 

Provider groups 

Center for Professional Development 

2,408 

1,251 

723 

57.8 

3 

3 

Institutions of higher education 

319 

307 

139 

45.3 

6 

5 

Regional education service agencies 

5,505 

4,126 

1,916 

46.4 

2 

2 

West Virginia Department of Education 

4,628 

3,732 

1,980 

53.1 

2 

1 

Individual provider 

Center for Professional Development 

2,408 

1,251 

723 

57.8 

3 

3 

Fairmont State University 

63 

63 

26 

41.3 

19 

13 

Marshall University 

256 

244 

113 

46.3 

7 

6 

RESA 1 

1,089 

841 

379 

45.1 

4 

3 

RESA2 

382 

346 

167 

48.3 

6 

5 

RESA 3 

1,374 

841 

347 

41.3 

5 

4 

RESA 4 

827 

578 

280 

48.4 

5 

4 

RESA 5 

441 

370 

196 

53.0 

5 

4 

RESA 6 

465 

418 

148 

35.4 

7 

6 

RESA 7 

683 

496 

276 

55.6 

5 

4 

RESA 8 

WVDE - Office of Assessment and 

244 

236 

123 

52.1 

6 

5 

Accountability 

WVDE - Office of Career and Technical 

402 

319 

162 

50.8 

6 

5 

Accountability and Support 

WVDE - Office of Career and Technical 

72 

72 

29 

40.3 

14 

12 

Instruction 

348 

266 

130 

48.9 

7 

6 

WVDE - Office of Early Learning 

190 

175 

84 

48.0 

8 

7 

WVDE - Office of Federal Programs 

WVDE - Office of Instructional 

11 

11 

9 

81.8 

15 

12 

Technology 

WVDE - Office of Professional 

2,564 

1,924 

993 

51.6 

2 

2 

Preparation (Certification) 

224 

206 

98 

47.6 

7 

6 

WVDE - Office of School Improvement 

95 

95 

40 

42.1 

12 

10 

WVDE - Office of Secondary Learning 

36 

36 

19 

52.8 

16 

13 

WVDE - Office of Special Programs 

686 

628 

416 

66.2 

3 

3 

Data Source: 2013-2014 PD Master Plan Session Report database 
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Appendix B. Additional Data Tables and Figures 


Table A 2. Number of Sessions in the PD Master Plan, and Number and Percent Delivered by Provider Group 
and Individual Provider 



Sessions in 

Sessions reported as 


PD Master 

delivered 


Provider in PD Master Plan 

Plan 

Number Percent 

Total 

479 

384 

80.2 

Provider groups 

Center for Professional Development 

65 

61 

93.8 

Institutions of higher education 

25 

15 

60.0 

Regional education service agencies 

156 

133 

85.3 

West Virginia Department of Education 

233 

175 

75.1 

Individual providers 

Center for Professional Development 

65 

61 

93.8 

Fairmont State University 

4 

4 

100.0 

Marshall University - Clinical Experiences and Professional Development 
Schools 

6 

2 

33.3 

Marshall University - June Harless Center 

15 

9 

60.0 

RESA 1 

49 

34 

69.4 

RESA2 

9 

8 

88.9 

RESA 3 

24 

20 

83.3 

RESA 4 

12 

9 

75.0 

RESA 5 

21 

21 

100.0 

RESA 6 

23 

23 

100.0 

RESA 7 

16 

16 

100.0 

RESA 8 

2 

2 

100.0 

WVDE - Office of Assessment and Accountability 

12 

11 

91.7 

WVDE - Office of Career and Technical Accountability and Support 

9 

4 

44.4 

WVDE - Office of Career and Technical Instruction 

60 

45 

75.0 

WVDE - Office of Early Learning 

7 

7 

100.0 

WVDE - Office of Federal Programs 

1 

1 

100.0 

WVDE - Office of Institutional Education Programs 

12 

8 

66.7 

WVDE - Office of Instructional Technology 

81 

63 

77.8 

WVDE - Office of Professional Preparation (Certification) 

13 

9 

69.2 

WVDE - Office of School Improvement 

6 

6 

100.0 

WVDE - Office of Secondary Learning 

21 

10 

47.6 

WVDE - Office of Special Programs 

11 

11 

100.0 

Data Source: 2013-2014 PD Master Plan Session Report database 
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Appendix B. Additional Data Tables and Figures 


Table A 3. Provider Performance in Submitting E-mail Addresses for Participants in Professional 
Development Sessions They Conducted 




Attendance 





reported 


Percentage of 



during first 


reported 


Attendance 

two data 

Number of 

participants 


reported all 

collection 

participant 

for whom 


three data 

periods (email 

email 

email 


collection 

addresses 

addresses 

addresses 

Provider 

periods 

required) 

provided 

were supplied 

TOTAL 

33,196 

27,271 

22,326 

67.8 

Provider groups 

Center for Professional Development 

3,785 

3,785 

3,718 

98.2 

Institutions of higher education 

762 

760 

552 

72.6 

Regional education service agencies 

11,289 

9,998 

9,376 

93.8 

West Virginia Department of 

Education 

17,360 

12,728 

8,680 

68.2 

Individual providers 

Center for Professional Development 

3,785 

3,785 

3,718 

98.2 

Fairmont State University 

172 

172 

160 

93.0 

Marshall University 

590 

588 

392 

66.7 

RESA 1 

2,242 

1,970 

1,834 

93.1 

RESA2 

754 

683 

683 

100.0 

RESA 3 

2,380 

2,048 

1,966 

96.0 

RESA 4 

1,910 

1,725 

1,609 

93.3 

RESA 5 

1,024 

908 

807 

88.9 

RESA 6 

1,052 

1,052 

923 

87.7 

RESA 7 

1,214 

1,099 

1,064 

96.8 

RESA 8 

713 

513 

490 

95.5 

WVDE Office of Assessment and 

737 

722 

652 

90.3 

Accountability 

WVDE Office of Career and Technical 

447 

447 

101 

22.6 

Accountability and Support 

WVDE Office of Career and Technical 

1,221 

1,174 

1,021 

87.0 

Instruction 

WVDE Office of Early Learning 

1,238 

504 

503 

99.8 

WVDE Office of Federal Programs 

28 

14 

14 

100.0 

WVDE Office of Institutional Education 

273 

273 

0 

0.0 

Programs 

WVDE Office of Instructional 

5,304 

4,460 

4,146 

93.0 

Technology 

WVDE Office of Professional 

1,419 

1,384 

376 

27.2 

Preparation (Certification) 

WVDE Office of School Improvement 

926 

896 

512 

57.1 

WVDE Office of Secondary Learning 

1,008 

956 

42 

4.4 

WVDE Office of Special Programs 

4,759 

1,898 

1,313 

69.2 

Data Source: 2013-2014 PD Master Plan Session Report database 
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Appendix B. Additional Data Tables and Figures 


Table A 4. Reasons Given for Not Implementing Some Sessions in the PD Master Plan by Provider 

Number Number not 
sessions implemented/ 

Organization/Reasons not implemented implemented not reported 

Center for Professional Development 61 3 

Board/department priorities changed (1) 

Other: (a) Session began and will continue into 2014-2015; (b) budget 
constraints (2) 

Marshall University - Clinical Experiences and Professional Development 

Schools 2 4 

Other: Staff member with this expertise left (1) 

Session was postponed to 2014-2015 (1) 

There were insufficient registrations (1) 

Topic was combined with another (1) 



Session was cancelled due to weather/water crisis (1) 
Session was postponed to 2014-2015 (1) 

There were insufficient registrations (2) 

Topic was combined with another (8) 

Topic was not requested (3) 


RESA 2 8 1 

No explanation provided (1) 

RESA 3 20 5 

No explanation provided (3) 

Session was postponed to 2014-2015 (2) 

RESA 4 10 2 

Topic was not requested (1) 

Session was delivered but not reported (1) 

Topic was combined with another (1) 

RESA 5 20 1 

Board/department priorities changed (1) 

WVDE - Office of Career and Technical Accountability and Support 4 5 

Other: (a) training handled on individual basis with schools through 
meetings; (b) staff member with this expertise retired. (2) 

Topic was combined with another (3) 

WVDE - Office of Career and Technical Instruction 48 12 


Other: Partner's responsibility as lead (1) 

No explanation provided (1) 

Session was delivered but not reported (7) 

Session was postponed to 2014-2015 (1) 

There were insufficient registrations (1) 

Topic was not requested (1) 

WVDE - Office of Institutional Education Programs 8 4 

Other: Two topics delivered by another provider; one topic handled on 
individual basis with schools through meetings (3) 

Session was delivered but not reported (1) 

Table continued on next page 
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Appendix B. Additional Data Tables and Figures 


Table A 4. Reasons Given for Not Implementing Some Sessions in the PD Master Plan by Provider 


Organization/Reasons not implemented 

Number 

sessions 

implemented 

Number not 
implemented/ 
not reported 

WVDE - Office of Instructional Technology 

63 

18 

No explanation provided (7) 

Other: Staff member with this expertise retired (1) 

Session was delivered but not reported (10) 



WVDE - Office of Professional Preparation (Certification) 

9 

4 


Other: (a) Topic mistakenly included in plan; (b) Work is ongoing, partly 
accomplished through other means (2) 

Session was postponed to 2014-2015 (1) 

Topic was combined with another (1) 

WVDE - Office of Secondary Learning 10 11 

Board/department priorities changed (1) 

Session was delivered but not reported (1) 

Topic was combined with another (2) 

Topic was not requested (7) 

Data Source: Agency staff response to e-mail query, September 2014. 


Table A 5. Provider Explanations for Sessions Not Delivered or Not Reported 

Explanation Number 

Total not delivered/not reported 93 

Canceled due to unforeseen circumstances 8 

• Board/department priorities changed (n = 3) 

• Budget constraints (n = 1) 

• Weather/water crisis (n = 1) 

• Staff member with this expertise left/retired (n = 3) 

Cancelled due to lack of interest 22 

• Topic not requested (n = 18) 

• Insufficient registrations (n = 4) 

Provider reporting error 14 

• No explanation provided (n = 12) 

• Mistakenly included in plan (n = 2) 

Provided (a) on different schedule, (b) by other means or providers, (c) provided but not 49 

reported 

• Session began and will continue into 2014-2015 (n = 1) 

• Session delivered but not reported (n = 20) 

• Session postponed to 2014-2015 (n = 6) 

• Topic delivered by another provider (n = 3) 

• Topic handled by other means or on individual basis (n = 3) 

• Topic combined with another (n = 16) 

Data Source: Agency staff response to e-mail query, September 2014. 
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Appendix B. Additional Data Tables and Figures 


Table A 6. Percent of Respondents That Reported Presence or Absence of Research-Based PD Practices by Role Group 


Content and content pedagogy focus 

Coherent with school/district goals and 
individuals' need for training 

Active learning 

Collective participation from own 
school/district 

Adequate duration and timespan 

Used 

Content Content curriculum 

knowledge instruction materials 

Challenging, 
Aligned with did not 
Goal training repeat other 

alignment needs PD** 

Discuss/ 

review Planning for 

student implementa Practice and 
work tion feedback 

Particip'n w/ Developed 

colleagues Motivated common 

from own collaboratio under- 
school n standings 

Enough Not too 

Enough experiences much time 

contact during for topic 

hours school year covered 

Role N % N % N % 

N % N % N % 

N % N % N % 

N % N % N % 

N % N % N % 


Principal/ Disagree 
assistant Agree 
principal NA 

Regular Disagree 
education Agree 
teacher NA 

Special Disagree 
education Agree 
teacher NA 


Library 

media 

specialist 

Instruct'! 

support 

teacher 


Disagree 

Agree 

NA 

Disagree 

Agree 

NA 


Other 


30 

499 
16 

191 

2101 

39 

29 

500 
11 

2 

39 

2 

10 

163 

4 


Parapro- Disagree 
fessional/ Agree 
aide NA 

District Disagree 
office Agree 

staff NA 

RESA staff Disagree 
Agree 
NA 


Disagree 

Agree 

NA 


5.5 

91.6 
2.9 

8.2 

90.1 

1.7 

5.4 

92.6 
2.0 

4.7 

90.7 
4.7 

5.6 

92.1 
2.3 


32 

366 

145 

210 

2010 

107 

41 

474 

24 

5 

34 

3 

19 

147 

11 


11 4.7 

219 94.4 

2 0.9 


5.9 

67.4 

26.7 


41 

297 

204 


11.9 

81.0 

7.1 

10.7 

83.1 

6.2 


24 10.3 

195 84.1 

13 5.6 


15 4.7 14 4.4 

296 91.9 191 59.9 

11 3.4 114 35.7 

1 2.5 2 5.0 

38 95.0 26 65.0 

1 2.5 12 30.0 

23 4.5 32 6.3 

460 90.6 380 74.7 

25 4.9 97 19.1 


7.6 

54.8 

37.6 


9.0 288 12.4 

86.4 1833 78.9 

4.6 203 8.7 


7.6 58 

87.9 436 

4.5 46 


5 

34 

4 

27 

123 

26 


10.7 

80.7 
8.5 

11.6 

79.1 

9.3 

15.3 

69.9 

14.8 


29 12.4 

182 78.1 

22 9.4 


16 

137 

166 


5.0 

42.9 

52.0 


3 7.7 

19 48.7 

17 43.6 

45 8.9 

324 63.9 

138 27.2 


SCHOOL-BASED RESPONDENTS 


25 

493 

24 


4.6 

91.0 

4.4 


79 

439 

21 


14.7 

81.4 

3.9 


156 

371 

15 


28.8 

68.5 

2.8 


139 6.0 360 15.6 588 25.4 

2103 90.6 1899 82.2 1634 70.6 

80 3.4 51 2.2 94 4.1 

43 8.0 63 11.7 155 28.7 

476 88.1 463 86.1 361 66.9 

21 3.9 12 2.2 24 4.4 


2 4.8 

39 92.9 

1 2.4 

8 4.5 

162 92.0 

6 3.4 


7 16.3 

35 81.4 

1 2.3 

34 19.4 

135 77.1 

6 3.4 


12 27.9 

30 69.8 

1 2.3 

46 26.0 

125 70.6 

6 3.4 


22 

192 

19 

12 

279 

28 


9.4 

82.4 
8.2 

3.8 

87.5 

8.8 


25 

205 

3 

43 

249 

24 


10.7 68 

88.0 154 

1.3 11 


29.2 

66.1 

4.7 


62 

410 

66 


11.5 

76.2 

12.3 


163 

316 

63 


30.1 

58.3 

11.6 


129 

360 

55 


23.7 

66.2 

10.1 


353 15.2 760 32.8 561 24.2 

1857 80.1 1404 60.5 1663 71.7 

107 4.6 155 6.7 95 4.1 

62 11.5 152 28.1 94 17.5 

454 84.1 348 64.3 422 78.4 

24 4.4 41 7.6 22 4.1 

7 16.3 13 31.0 10 23.3 

32 74.4 20 47.6 30 69.8 

4 9.3 9 21.4 3 7.0 

29 16.5 53 29.9 39 22.3 

137 77.8 108 61.0 124 70.9 

10 5.7 16 9.0 12 6.9 


16 

208 

9 


6.9 
89.3 

3.9 


57 

151 

25 


24.5 25 10.6 

64.8 200 85.1 

10.7 10 4.3 


NON-SCHOOL-BASED AND OTHER RESPONDENTS 


13.6 71 

78.8 222 

7.6 22 


22.5 

70.5 
7.0 


2 5.0 

31 77.5 

7 17.5 

27 5.4 

410 81.5 

66 13.1 


6 15.0 

34 85.0 

0 0.0 


10 25.0 

30 75.0 

0.0 


70 13.9 94 18.5 

396 78.6 369 72.6 

38 7.5 45 8.9 


30 

235 

53 


9.4 

73.9 

16.7 


66 

177 

72 


21.0 

56.2 

22.9 


69 

177 

69 


21.9 

56.2 

21.9 


5 12.8 5 12.8 6 15.4 

28 71.8 26 66.7 27 69.2 

6 15.4 8 20.5 6 15.4 

66 12.9 126 24.8 88 17.3 

391 76.7 287 56.4 336 66.0 

53 10.4 96 18.9 85 16.7 


31 

484 

30 


5.7 

88.8 

5.5 


94 

386 

64 


17.3 

71.0 

11.8 


70 

411 

63 


10 23.3 

31 72.1 

2 4.7 


11 25.6 

30 69.8 

2 4.7 


13 7.3 27 15.3 

151 85.3 140 79.1 

13 7.3 10 5.6 


34 14.6 

185 79.4 

14 6.0 

13 4.1 

267 84.5 

36 11.4 

4 10.3 

27 69.2 

8 20.5 

53 10.5 

393 77.5 

61 12.0 


41 17.5 

170 72.6 

23 9.8 


2 

27 

10 


5.1 

69.2 

25.6 


12.9 

75.6 

11.6 


253 10.9 451 19.4 385 16.6 

1942 83.6 1689 72.7 1752 75.6 

128 5.5 184 7.9 181 7.8 

63 11.7 87 16.1 72 13.3 

445 82.4 400 73.9 413 76.5 

32 5.9 54 10.0 55 10.2 


6 14.0 

36 83.7 

1 2.3 

28 15.9 

134 76.1 

14 8.0 


34 

179 

19 


32 10.1 23 

218 68.8 242 

67 21.1 52 


2 

29 

8 


14.7 

77.2 
8.2 

7.3 

76.3 

16.4 

5.1 

74.4 

20.5 


72 14.1 61 12.0 

350 68.8 360 70.7 

87 17.1 88 17.3 


97 

425 

21 


17.9 

78.3 

3.9 


111 

364 

70 


454 19.6 585 

1802 77.9 1538 

58 2.5 187 


20.4 141 26.1 

66.8 367 67.8 

12.8 33 6.1 

25.3 512 22.0 

66.6 1705 73.4 

8.1 105 4.5 


82 15.3 108 20.1 139 25.9 

440 82.1 384 71.4 368 68.5 

14 2.6 46 8.6 30 5.6 

12 27.9 12 27.9 10 23.3 

31 72.1 27 62.8 29 67.4 

0.0 4 9.3 4 9.3 

35 19.9 50 28.4 30 16.9 

131 74.4 114 64.8 140 79.1 

10 5.7 12 6.8 7 4.0 


21 9.0 

206 88.4 

6 2.6 


51 

233 

30 


16.2 

74.2 

9.6 


35 

179 

18 

47 

185 

84 


15.1 97 41.6 

77.2 122 52.4 

7.8 14 6.0 


14.9 

58.5 

26.6 


58 18.4 
228 72.2 
30 9.5 


2 5.1 5 13.5 7 17.9 

33 84.6 23 62.2 30 76.9 

4 10.3 9 24.3 2 5.1 

77 15.2 93 18.3 84 16.5 

391 77.0 299 58.9 381 75.0 

40 7.9 116 22.8 43 8.5 


Notes: Highlighted cells indicate that 15% or more of respondents considered the research-based PD practice "not applicable." 

*Disagree = disagree and strongly disagree; Agree = Agree and strongly disagree; NA = not applicable. Data source: WVBE PD Master Plan Participant Survey 2013-2014 
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Principal/assistant principal (n = 545) 
Regular education teacher (n = 2,331) 
Special education teacher (n = 540) 
Library media specialist (n = 43) 
Instructional support teacher (n = 177) 
Paraprofessional/aide (n = 232) 

District central office staff (n = 322) 
RESA staff (n = 40) 
Other (n = 508) 


10.93.6 17.3 53.9 

14.3 



5.34.5 19.4 51.2 

19.7 




5.64.8 16.0 55.3 

18.2 



6.4 7.3 17.8 49.4 

19.2 


6.24.4 19.7 50.5 

19.2 


6.CB.5 15.1 56.5 

19.0 



18.0 2.9 15.7 47.3 

16.1 


16.7 4.8 13.1 38.8 

26.7 


14.2 4.8 16.0 46.6 18.5 


0% 20% 40% 60% 80% 100% 

Percent of respondents 


■ Not applicable ■ Strongly disagree ■ Disagree ■ Agree ■ Strongly agree 


Figure A 1. Average Rate of Agreement or Disagreement That Professional Development Attended Adhered 
to Research-Based Practices, by Role Group. 

The top six bars represent role groups based in schools, while the lower three bars are non-school and 
"other" role groups. The two major groups differed in how they rated the overall alignment of the 
professional development each received with research-based PD practices-measures that tend to be more 
pertinent for teachers and others working directly with students. Paraprofessionals, special education 
teachers, regular education teachers (in descending order) were most in agreement that the session they 
attended had the five qualities of research-based practice. Data Source: School-based respondents to 
WVBE PD Master Plan Participant Survey 2013-2014 
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Table A 7. Percent of School-Based Respondents That Agreed or Strongly Agreed Five Research-Based PD 
Practices Were Present in the Sessions They Attended, by Individual Provider 



Average percent of agree/strongly agreed 



Provider 

All research- 

based PD 
practices 

Content 

focus Coherence 

Active Collective 

learning participation 

Duration/ 

timespan 

Total (n = 3,879) 

82.0 

90.3 

83.3 

75.8 

84.1 

76.4 

WVDE - Office of Career 

and Technical 
Accountability and 
Support (n = 27) 

89.6 

96.2 

91.0 

91.7 

88.5 

80.8 

Marshall University - June 
Harless Center (n = 

76) 

85.6 

90.7 

86.0 

83.1 

89.0 

79.2 

WVDE - Office of 

Instructional 
Technology (n = 850) 

85.2 

92.8 

85.5 

84.5 

82.2 

80.9 

RESA 1 (n = 292) 

85.0 

92.0 

84.2 

79.8 

90.7 

78.3 

Fairmont State University 
(n = 26 

84.1 

86.3 

84.2 

79.6 

88.3 

81.8 

WVDE - Office of Special 
Programs (n = 296) 

84.1 

94.2 

85.3 

77.6 

83.8 

79.6 

Center for Professional 
Development (n = 

638) 

83.7 

90.1 

85.3 

78.3 

84.2 

80.6 

RESA 6 (n = 132) 

82.9 

92.1 

86.6 

73.0 

89.0 

73.9 

RESA 4 (n = 229) 

81.8 

91.5 

82.2 

75.8 

86.5 

73.0 

RESA 2 (n = 139) 

81.2 

90.3 

82.6 

70.0 

87.7 

75.6 

WVDE - Office of School 
Improvement (n = 23) 

81.0 

93.8 

84.1 

62.7 

86.0 

78.3 

WVDE - Office of Career 

and Technical 
Instruction (n = 115) 

79.1 

87.4 

83.2 

71.3 

81.0 

72.5 

WVDE - Office of 

Assessment and 
Accountability (n = 93) 

77.4 

86.2 

80.6 

66.2 

82.9 

71.0 

RESA 5 (n = 188) 

77.2 

88.2 

81.4 

67.8 

81.9 

66.7 

RESA 3 (n = 207) 

76.2 

84.3 

77.2 

66.9 

84.2 

68.6 

RESA 7 (n = 234) 

75.1 

83.3 

77.9 

63.8 

80.8 

69.7 

RESA 8 (n = 104) 

74.9 

86.7 

77.9 

59.9 

75.8 

74.0 

WVDE - Office of 

Professional 
Preparation 
(Certification) (n = 70) 

74.9 

83.7 

76.5 

59.3 

85.6 

69.4 

Note: Analysis based on responses from school-based respondents only. Four providers, Marshall University's 

Department of Clinical Studies and Professional Development Schools and the WVDE Offices of Federal Pro- 

grams, Early Learning, and Secondary Learning were not included because there were fewer than 20 respond- 
ents for these items among the school-based respondents. 

Data source: WVBE PD Master Plan Participant Survey 2013-2014 
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All (n = 33,196) 
CPD (n = 3,785) 
FSU (n = 172) 
MU Clinical (n = 72) 
MU JHC (n = 518) 
RESA 1 (n = 2,242) 
RESA 2 (n = 754) 
RESA 3 (n = 2,380) 
RESA 4 (n = 1,910) 
RESA 5 (n = 1,024) 
RESA 6 (n = 1,052) 
RESA 7 (n = 1,214) 
RESA 8 (n = 713) 
WVDE OAA (n = 737) 
WVDE OCTAS (n = 447) 
WVDE OCTI (n = 1,221) 
WVDE OEL(n = 1,238) 
WVDE OFP (n = 28) 
WVDE OIEP (n = 273) 
WVDE OIT (n = 5,304) 
WVDE OPP (n = 1,419) 
WVDE OSI (n = 926) 
WVDE OSL (n = 1,008) 
WVDE OSP(n = 4,759) 


27.6 37.1 17.0 18.3 


70.4 29.6 


64.5 35.5 


52.8 47.2 


13.7 34.7 20.5 31.1 


50.4 42.3 7.3 


24.8 75.2 


48.6 45.5 5.9 


50.8 36.2 10.9 2.0 


13.6 5.2 77.8 3.4 


10.5 35.0 53.4 1.1 


26.6 63.8 | 9.6 


37.2 62.8 


68.7 24.8 6.5 


53.9 31.5 14.5 


00 

i-> 

Cn 

LD 

00 

oo 

o 

XD~ 


40.4 35.5 24.1 


100.0 


92.3 7.7 


12.5 8.1 0.6 [ 78.8 


81.7 18.3 


97.5 2.5 


88.1 11.9 


13.0 19.6 57.7 9.7 


0 10 
1 to 4 hours 


20 


30 


60 


70 


80 


40 50 

Percent of participants for each duration category 
5 to 13 hours ■ 14 to 29 hours ■ 30 or more hours 


90 


100 


Figure A 2. Duration of Professional Development by Individual Provider 
Data Source: 2013-2014 PD Master Plan Session Report database 

All = All providers; CPD = Center for Professional Development; FSU = Fairmont State University; MU Clinical 
= Marshall University Clinical Experiences & PD Schools; MU J HC = Marshall University June Harless Center; 
RESA = regional education service agency (1-8); WVDE OAA = Office of Assessment and Accountability; 
WVDE OCTAS = Office of Career and Technical Accountability and Support; WVDE OCTI = Office of Career 
and Technical Instruction; WVDE OEL = Office of Early Learning; WVDE OFP = Office of Federal Programs; 
WVDE OIEP = Office of Institutional Education Programs; WVDE OIT = Office of Instructional Technology; 
WVDE OPP = Office of Professional Preparation; WVDE OSI = Office of School Improvement; WVDE OSL = 
Office of Secondary Learning; WVDE OSP = Office of Special Programs 
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All (n = 33,196) 
CPD(n = 3,785) 
FSU (n = 172) 
MU Clinical (n = 72) 
MU JHC (n = 518) 
RESA 1 (n = 2,242) 
RESA 2 (n = 754) 
RESA 3 (n = 2,380) 
RESA 4 (n = 1,910) 
RESA 5 (n = 1,024) 
RESA 6 (n = 1,052) 
RESA 7 (n = 1,214) 
RESA 8 (n = 713) 
WVDE OAA (n = 737) 
WVDE OCTAS (n = 447) 
WVDE OCTI (n = 1,221) 
WVDE OEL(n = 1,238) 
WVDE OFP (n = 28) 
WVDE OIEP (n = 273) 
WVDE OIT(n = 5,304) 
WVDE OPP (n = 1,419) 
WVDE OSI (n = 926) 
WVDE OSL (n = 1,008) 
WVDE OSP(n = 4,759) 


50.6 22.9 5.0 13.0 8.5 


48.8 35.0 | 16.2 


76.7 11.6 11.6 


52.8 47.2 


20.8 26.8 4.1 48.3 


91.7 5.7 2.0 j) 


100.0 


90.5 8.5 0|e 


81.1 13.3 1.2 3. 


18.8 62.9 6.4 6.1 5.9 | 


29.5 51.8 15.2 2.6 l.|o 


53.2 31.1 2.7 12.9 


98.2 1.8 


79.9 11.1 0.0 9.0 


53.9 24.6 14.5 6.9 


89.6 7.5 0.9 2 


1.9 14.1 40.4 43.5 


100.0 


100.0 


20.6 1.4 75.9 2.1 


37.1 4.0 58.9 


43.3 56.7 


34.1 38.4 27.5 


35.5 53.4 11.0 


0 

1 day 


10 


20 


30 


40 


50 


60 


70 


80 


90 


100 


Percent of participants for each timespan category 
2 - 7 days «8- 30 days ■ 31 -60 days ■ More than 60 days 


Figure A 3. Timespan of Professional Development by Individual Provider 
Data Source: 2013-2014 PD Master Plan Session Report database 

All = All providers; CPD = Center for Professional Development; FSU = Fairmont State University; MU Clinical 
= Marshall University Clinical Experiences & PD Schools; MU JHC = Marshall University June Harless Center; 
RESA = regional education service agency (1-8); WVDE OAA = Office of Assessment and Accountability; 
WVDE OCTAS = Office of Career and Technical Accountability and Support; WVDE OCTI = Office of Career 
and Technical Instruction; WVDE OEL = Office of Early Learning; WVDE OFP = Office of Federal Programs; 
WVDE OIEP = Office of Institutional Education Programs; WVDE OIT = Office of Instructional Technology; 
WVDE OPP = Office of Professional Preparation; WVDE OSI = Office of School Improvement; WVDE OSL = 
Office of Secondary Learning; WVDE OSP = Office of Special Programs 
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Table A 8. Participants' Views About Helpfulness of the Session in Meeting the Targeted Board Goal 


Helpful in 
meeting aligned 
goal 


Goal 1. Goal 2. Goal 3. Goal 4. 

Comprehensive Early Content & Pedagogy Leadership Educator Evaluation 

Literacy to Standards Competencies System 

Frequency Percent Frequency Percent Frequency Percent Frequency Percent 


Total 

374 

100.0 

3,063 

100.0 

561 

100.0 

414 

100.0 

Not applicable 

32 

8.6 

362 

11.8 

76 

13.5 

21 

5.1 

Strongly disagree 

4 

1.1 

52 

1.7 

7 

1.2 

5 

1.2 

Disagree 

29 

7.8 

342 

11.2 

46 

8.2 

33 

8.0 

Agree 

219 

58.6 

1,879 

61.3 

342 

61.0 

257 

62.1 

Strongly agree 

90 

24.1 

428 

14.0 

90 

16.0 

98 

23.7 


Data source: WVBE PD Master Plan Participant Survey 2013-2014 
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Table A 9. Perceived Impact of Professional Development (Pre/Post) Overall, by Provider Group, and by 
Individual Provider: Statistical Significance and Effect Size 

Paired Differences 

95% Confidence 
Std. Er- Interval of the Dif- 


Pre/post pairs 

N 

Std. Devi- 

Mean ation 

ror 

Mean 

ference 

Lower Upper 

t 

df 

Sig. Cohen's 
(2-tail) d 

All providers 

Knowledge 

4,358 

.66223 

.73921 

.01120 

.64028 

.68418 

59.140 

4357 

.000 

1.0 

Practice of skills 

4,153 

.48327 

.73599 

.01142 

.46087 

.50566 

42.315 

4152 

.000 

0.6 

Attitudes/beliefs 

4,201 

.31516 

.62578 

.00965 

.29623 

.33409 

32.643 

4200 

.000 

0.4 




Provider groups 








Center for Professional Development 





Knowledge 

687 

.72198 

.74221 

.02832 

.66638 

.77758 

25.496 

686 

.000 

1.1 

Practice of skills 

656 

.55335 

.74969 

.02927 

.49588 

.61083 

18.905 

655 

.000 

0.8 

Attitudes/beliefs 

657 

.30289 

.60577 

.02363 

.25649 

.34930 

12.816 

656 

.000 

0.4 

Institutions of higher education 

Knowledge 

130 

1.00000 

.83527 

.07326 

.85506 

1.14494 

13.650 

129 

.000 

1.3 

Practice of skills 

123 

.61789 

.84466 

.07616 

.46712 

.76865 

8.113 

122 

.000 

0.7 

Attitudes/beliefs 

120 

.45833 

.72060 

.06578 

.32808 

.58859 

6.968 

119 

.000 

0.6 

Regional education service agencies 

Knowledge 

1,822 

.60593 

.73099 

.01713 

.57234 

.63951 

35.382 

1821 

.000 

0.9 

Practice of skills 

1,758 

.44425 

.72599 

.01731 

.41029 

.47821 

25.657 

1757 

.000 

0.6 

Attitudes/beliefs 

1,764 

.29422 

.61569 

.01466 

.26547 

.32297 

20.070 

1763 

.000 

0.4 

West Virginia Department of Education 

Knowledge 

1,884 

.68471 

.73481 

.01693 

.65151 

.71792 

40.446 

1883 

.000 

1.0 

Practice of skills 

1,775 

.50423 

.73168 

.01737 

.47016 

.53829 

29.034 

1774 

.000 

0.7 

Attitudes/beliefs 

1,821 

.34871 

.63832 

.01496 

.31937 

.37805 

23.312 

1820 

.000 

0.4 




Individual providers 








Center for Professional Development 





Knowledge 

681 

.72540 

.73929 

.02833 

.66978 

.78103 

25.606 

680 

.000 

1.1 

Practice of skills 

651 

.55607 

.74905 

.02936 

.49842 

.61372 

18.941 

650 

.000 

0.8 

Attitudes/beliefs 

652 

.30368 

.60701 

.02377 

.25700 

.35036 

12.774 

651 

.000 

0.4 

Fairmont State University 

Knowledge 

26 

.57692 

.64331 

.12616 

.31709 

.83676 

4.573 

25 

.000 

0.9 

Practice of skills 

25 

.40000 

.64550 

.12910 

.13355 

.66645 

3.098 

24 

.005 

0.8 

Attitudes/beliefs 

26 

.26923 

.53349 

.10463 

.05375 

.48471 

2.573 

25 

.016 

0.5 

Marshall University - Clinical Experiences and Professional Development Schools 

Knowledge 

22 

.77273 

.61193 

.13046 

.50141 

1.04404 

5.923 

21 

.000 

1.6 

Practice of skills 

22 

.68182 

.83873 

.17882 

.30995 

1.05369 

3.813 

21 

.001 

0.9 

Attitudes/beliefs 

22 

.54545 

.59580 

.12703 

.28129 

.80962 

4.294 

21 

.000 

1.0 

Marshall University - June Harless Center 

Knowledge 

78 

1.16667 

.88884 

.10064 

.96626 

1.36707 

11.592 

77 

.000 

1.6 

Practice of skills 

73 

.65753 

.90091 

.10544 

.44734 

.86773 

6.236 

72 

.000 

0.7 

Attitudes/beliefs 

68 

.52941 

.81900 

.09932 

.33117 

.72765 

5.330 

67 

.000 

0.6 

RESA1 

Knowledge 

354 

.64689 

.78380 

.04166 

.56496 

.72882 

15.529 

353 

.000 

1.0 

Practice of skills 

336 

.38393 

.77935 

.04252 

.30029 

.46756 

9.030 

335 

.000 

0.5 

Attitudes/beliefs 

342 

.23977 

.67711 

.03661 

.16775 

.31178 

6.549 

341 

.000 

0.3 


Table A 9 continues on next page. 
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Table A 9. Perceived Impact of Professional Development (Pre/Post) Overall, by Provider Group, and by 
Individual Provider: Statistical Significance and Effect Size 

Paired Differences 

95% Confidence 
Std. Er- Interval of the Dif- 


Pre/post pairs 

N 

Std. Devi- 

Mean ation 

ror 

Mean 

ference 

Lower Upper 

t 

df 

Sig. Cohen's 
(2-tail) d 

RESA2 

Knowledge 

159 

.54088 

.66315 

.05259 

.43701 

.64475 

10.285 

158 

.000 

1.0 

Practice of skills 

156 

.50000 

.71392 

.05716 

.38709 

.61291 

8.748 

155 

.000 

0.7 

Attitudes/beliefs 

159 

.30818 

.64582 

.05122 

.20702 

.40933 

6.017 

158 

.000 

0.4 

RESA3 

Knowledge 

325 

.56923 

.70643 

.03919 

.49214 

.64632 

14.526 

324 

.000 

0.8 

Practice of skills 

314 

.40764 

.67786 

.03825 

.33238 

.48291 

10.656 

313 

.000 

0.5 

Attitudes/beliefs 

313 

.27157 

.57712 

.03262 

.20738 

.33575 

8.325 

312 

.000 

0.3 

RESA4 

Knowledge 

263 

.53612 

.64621 

.03985 

.45766 

.61458 

13.455 

262 

.000 

0.8 

Practice of skills 

255 

.46275 

.69700 

.04365 

.37679 

.54870 

10.602 

254 

.000 

0.6 

Attitudes/beliefs 

252 

.36905 

.62701 

.03950 

.29126 

.44684 

9.343 

251 

.000 

0.5 

RESA5 

Knowledge 

187 

.65775 

.74810 

.05471 

.54983 

.76568 

12.023 

186 

.000 

1.1 

Practice of skills 

181 

.47514 

.69577 

.05172 

.37309 

.57719 

9.187 

180 

.000 

0.6 

Attitudes/beliefs 

183 

.29508 

.58397 

.04317 

.20991 

.38026 

6.836 

182 

.000 

0.3 

RESA6 

Knowledge 

139 

.73381 

.76685 

.06504 

.60520 

.86242 

11.282 

138 

.000 

1.2 

Practice of skills 

135 

.54074 

.74074 

.06375 

.41465 

.66683 

8.482 

134 

.000 

0.8 

Attitudes/beliefs 

134 

.32090 

.60822 

.05254 

.21697 

.42482 

6.107 

133 

.000 

0.4 

RESA7 

Knowledge 

263 

.69582 

.79037 

.04874 

.59985 

.79178 

14.277 

262 

.000 

1.0 

Practice of skills 

253 

.46245 

.80399 

.05055 

.36290 

.56200 

9.149 

252 

.000 

0.6 

Attitudes/beliefs 

251 

.32669 

.61061 

.03854 

.25079 

.40260 

8.476 

250 

.000 

0.4 

RESA8 

Knowledge 

120 

.38333 

.63753 

.05820 

.26810 

.49857 

6.587 

119 

.000 

0.6 

Practice of skills 

117 

.40171 

.61672 

.05702 

.28878 

.51464 

7.046 

116 

.000 

0.5 

Attitudes/beliefs 

119 

.21008 

.50210 

.04603 

.11894 

.30123 

4.564 

118 

.000 

0.2 



WVDE 

- Office of Assessment and Accountability 




Knowledge 

153 

.58824 

.61269 

.04953 

.49037 

.68610 

11.876 

152 

.000 

0.8 

Practice of skills 

140 

.35714 

.57616 

.04869 

.26086 

.45342 

7.334 

139 

.000 

0.3 

Attitudes/beliefs 

145 

.19310 

.53094 

.04409 

.10595 

.28025 

4.380 

144 

.000 

0.2 

WVDE - Office of Career and Technical Accountability and Support 

Knowledge 

28 

1.07143 

1.01575 

.19196 

.67756 

1.46529 

5.582 

27 

.000 

2.0 

Practice of skills 

26 

.53846 

.70602 

.13846 

.25329 

.82363 

3.889 

25 

.001 

0.8 

Attitudes/beliefs 

28 

.28571 

.59982 

.11336 

.05313 

.51830 

2.521 

27 

.018 

0.3 



WVDE - 

Office of Career and Technical Instruction 




Knowledge 

114 

.51754 

.74350 

.06963 

.37958 

.65550 

7.432 

113 

.000 

0.8 

Practice of skills 

103 

.37864 

.72917 

.07185 

.23613 

.52115 

5.270 

102 

.000 

0.5 

Attitudes/beliefs 

109 

.18349 

.49376 

.04729 

.08974 

.27723 

3.880 

108 

.000 

0.2 




WVDE - 

Office of Early Learning 





Knowledge 

77 

.31169 

.54434 

.06203 

.18814 

.43524 

5.025 

76 

.000 

0.4 

Practice of skills 

65 

.13846 

.46358 

.05750 

.02359 

.25333 

2.408 

64 

.019 

0.1 

Attitudes/beliefs 

74 

.08108 

.43025 

.05002 

-.01860 

.18076 

1.621 

73 

.109* 

0.1 


Table A 9 continues on next page. 
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Appendix B. Additional Data Tables and Figures 


Table A 9. Perceived Impact of Professional Development (Pre/Post) Overall, by Provider Group, and by 
Individual Provider: Statistical Significance and Effect Size 

Paired Differences 

95% Confidence 
Std. Er- Interval of the Dif- 

Std. Devi- ror ference Sig. Cohen's 

Pre/post pairs N Mean ation Mean Lower Upper t df (2-tail) d_ 

WVDE - Office of Federal Programs 

Knowledge 8 .75000 .70711 .25000 .15884 1.34116 3.000 7 .020 1.6 

Practice of skills 8 .50000 .53452 .18898 .05313 .94687 2.646 7 .033 0.5 

Attitudes/beliefs 8 .37500 .51755 .18298 -.05768 .80768 2.049 7 .080* 0.8 

WVDE - Office of Instructional Technology 


Knowledge 

947 

.75607 

.76156 

.02475 

.70751 

.80464 

30.552 

946 

.000 

1.2 

Practice of skills 

916 

.52183 

.76434 

.02525 

.47227 

.57140 

20.663 

915 

.000 

0.7 

Attitudes/beliefs 

921 

.39522 

.67578 

.02227 

.35152 

.43892 

17.749 

920 

.000 

0.5 

WVDE - Office of Professional Preparation (Certification) 

Knowledge 

89 

.71910 

.76854 

.08147 

.55721 

.88100 

8.827 

88 

.000 

0.9 

Practice of skills 

80 

.52500 

.81092 

.09066 

.34454 

.70546 

5.791 

79 

.000 

0.7 

Attitudes/beliefs 

78 

.38462 

.60797 

.06884 

.24754 

.52169 

5.587 

77 

.000 

0.5 

WVDE - Office of School Improvement 

Knowledge 

37 

.48649 

.73112 

.12020 

.24272 

.73025 

4.047 

36 

.000 

0.8 

Practice of skills 

34 

.41176 

.60891 

.10443 

.19931 

.62422 

3.943 

33 

.000 

0.6 

Attitudes/beliefs 

36 

.11111 

.52251 

.08708 

-.06568 

.28790 

1.276 

35 

.210* 

0.2 

WVDE - Office of Secondary Learning 

Knowledge 

19 

.47368 

.77233 

.17718 

.10143 

.84593 

2.673 

18 

.016 

0.6 

Practice of skills 

19 

.57895 

.60698 

.13925 

.28639 

.87150 

4.158 

18 

.001 

0.7 

Attitudes/beliefs 

18 

.22222 

.73208 

.17255 

-.14183 

.58628 

1.288 

17 

.215* 

0.3 

WVDE - Office of Special Programs 

Knowledge 

269 

.58736 

.60839 

.03709 

.51433 

.66039 

15.834 

268 

.000 

0.9 

Practice of skills 

244 

.55328 

.70945 

.04542 

.46382 

.64274 

12.182 

243 

.000 

0.8 

Attitudes/beliefs 

263 

.34601 

.62246 

.03838 

.27043 

.42158 

9.015 

262 

.000 

0.4 


Data source: WVBE PD Master Plan Participant Survey 2013-2014 


48 I Implementation of the Master Plan for Statewide Professional Staff Development for 2013-2014 








ONEVOICe 
ON EFOCUS 

All Students Achieving 


ifWaiVt 


LhXAy Department of 


Michael J. Martirano Ed.D. 
State Superintendent of Schools 


